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Abstract 

Abstract 

A policy  study  was  conducted  in  four  rounds  to  assess  the  anticipated  payoff  of  an 
investment  by  ARPA  in  the  development  of  a new  family  of  terminals  for  use  by  the 
military  in  computer-assisted  and  computer-managed  instruction.  Members  of  the  panel  of 
experts  (11  civilian,  9 military)  proposed  features,  rated  them,  and  reacted  to  the  resulting 
ratings.  Of  the  24  features,  those  rated  as  most  needed  were  seen  by  these  experts  as 
likely  to  be  in  commercial  production  five  to  ten  years  from  now  in  a form  usoble  by  the 
military.  Experts  generally  agreed  that  investments  in  innovative  pedagogical  software 
and  in  innovative  coursewriting  are  likely  to  have  a greater  payoff  than  an  investment  in 
terminal  development.  Of  14  software  features,  those  rated  as  most  needed  are  ones  for 
particularizing  instruction  online  to  the  course-related  needs  of  individual  students. 

The  report  is  a thorough  description  of  the  conduct  of  the  study.* 


♦ A companion  report  by  Louis  Gallenson,  ^ Approach  ^ Providing  a User  Interface  f^ 
Military  Computer-Aided  Instruction  m 1980.  ISI/RR-75-43,  discusses  effective  utilization  of 
forthcoming  commercial  terminals  in  military  computer-aided  instruction. 
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Executive  Summary 

Executive  Summary 

- A panel  of  experts  (11  civilian  and  9 military)  was  selected  from  the 
computer-assisted  and  computer-managed  instruction  (CAI/CMI)  community. 

- Panel  members  were  polled  in  order  to  determine  what  features  they  felt  should 
go  into  innovative  military  CAI/CMI  terminals  and  what  payoffs  they  anticipated  would 
follow  from  these  features. 

- The  four  rounds  of  questionnaire  and  feedback  included:  (1)  general  stance 
probing,  (2)  open-ended  soliciting  of  features  and  payoffs,  (3)  ranking  of  features  and 

payoffs,  and  (4)  eliciting  reactions  to  resulting  rankings. 

- Participants  rated  investments  m either  innovative  pedagogic  software  pL 
innovative  coursewriting  as  haying  hist^  PotentjaL  payoffs  than  investmej^  in  t^minal 

technology  or  large-scale  use  of  existing  systems^ 

- The  features  felt  to  be  most  necessary  in  all  CAI/CMI  terminals  are  likely  to  be  in 
commercial  production  by  mainstream  terminal  vendors  in  a form  usable  by  the  military  by 
1980-1985. 

- With  respect  to  pedagogical  software,  the  features  felt  to  be  most  necessary  are 
those  that  facilitate  tailoring  of  interaction  with  a student  to  his  particular  course-related 
needs,  interests,  and  difficulties. 

The  report  is  a thorough  description  of  the  conduct  of  the  study. 


This  report  deals  with  one  of  many  activities  in  a project  attempting  to  provide 
functional  specifications  for  a family  of  new  CAI  (Computer-Assisted  Instruction)  and  CMI 
(Computer  Managed  Instruction)  terminals  for  the  1980-1985.  The  purpose  of  the  activity 
was  to  survey  CAI/CMI  experts  in  order  to  determine  what  features  they  felt  should  go 
into  innovative  military  CAI/CMI  teiminals. 


The  rationale  for  the  survey  was  that  innovative  terminal  design  could  be  informed 
by  sampling  the  projected  needs  and  uses  of  the  CAI/CMI  user  and  research  communilies. 
It  was  assumed  at  the  outset  of  the  study  that  representatives  of  these  communities  could 
provide  descriptions  not  only  of  desired  terminal  features,  but  also  of  anticipated  payoffs 
from  use  of  these  features. 


The  goals  for  the  querying  were  to 


1.  (1)  Discover  what  features  are  desired  in  the  military  CAI/CMI  terminal  (or  family 
of  terminals)  in  1980-1985. 


2.  (2)  Determine  what  the  anticipated  pa/offs  are  for  implementing  and  using  these 
features. 


3.  (3)  Determine  the  importance  of  military  CAI/CMI  terminal  development  relative 
to  developments  in  software,  large-scale  demonstrations,  etc. 


A.  (A)  Determine  the  projected  value  of  stand-alone  versus  networked  systems. 


In  order  to  carry  out  the  survey,  a panel  of  experts  was  selected  by  the  study 
group  working  in  conjunction  with  the  ARPA  Project  Officer.  The  panel  members  were 
surveyed  using  an  iterative  questionnaire  technique  described  below.  The  procedures  and 
materials  used  in  the  study  are  also  described  in  a separate  section  below.  Finally,  the 
conclusions  and  recommendations  resulting  from  the  survey  are  repotted. 


m 
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Problems  of  Need  Assessment 


Modern  technology  is  best  characterized  as  immensely  complicated,  expensive  to 
develop,  and  requiring  long  lead  times  from  initial  specification  to  mass  production.  Often 
the  cost  of  developing  a new  device  is  so  high  that  potential  users  can  afford  it  only  if 
they  are  willin'  to  guarantee  a certain  minimum  order  to  the  producer.  Even  without  an 
agreement,  the  market  ceases  to  act  as  a testing  ground  for  determining  user  needs  when 
demand  is  low  relative  to  high  initial  development  costs.  Customers  are  forced  to  take 
what  is  available  and  like  it.  With  long  lead  times  and  many  innovative  ideas  that  cannot 
be  pursued  profitably,  it  is  not  surprising  that  producers  often  fail  to  perceive  the  market 
correctly.  It  is  in  the  best  interests  of  all  parties  that  representatives  of  the  potential 
user  communities  be  involved  in  specifying  the  needs  to  which  new  technologies  are 
supposed  to  respond. 

This  report  describes  a methodology  similar  to  Turoff’s  Policy  Delphi  which  user 
communities  mi[  ht  employ  when  establishing  their  needs.*  It  has  been  refined  during  a 
study  of  features  of  computer  terminals  necessary  for  CAI  or  CMI. 

Before  attempting  a need  assessment  study,  one  should  understand  what  is  invoiv^. 
The  task  is  amorphous.  Perfly,  it  is  an  attempt  to  define  the  appropriate  community. 
Partly,  H ts  an  attempt  to  dlecoww  the  Jlrecltars  thst  the  technolo^^y  is  taking.  One  hopes 
by  shifting  back  and  forth  to  discover  the  needs  of  a set  of  people  which  it  appears  to  be 
feasible  and  extremely  beneficial  to  satisfy.  One  can  rarely  go  into  the  marketplace  and 
survey  needs,  because  consumers  do  not  Know  what  is  or  is  not  achievable  by  tecirmolcgy. 
In  rapidly  evolving  technological  fields,  even  those  professionally  involved  have  difficulty 
ec5iK;ci-''.g  w^sl  i«  ac^ievatlo  in  s givflrt  time  frame.  An  effort  to  determine  needs  is  likely 
to  be  as  much  an  educational  process  as  it  is  an  opinion-probing  process.  One  of  the 
greatest  dangers  is  in  assuming  too  much.  User  representatives  are  likely  to  possess 
widely  differing  competencies,  interests,  and  skills.  While  there  might  be  some  advantage 
to  having  a ranJomiy  selected  set  of  indiviJliaft  rank  order  their  nee  is  il  sficms  most 
reasonable  that  the  user  community  representatives  should  be  careful'y  selected  for  their 
knowledge  and  commitment.  They  should  be  allowed  to  interact  over  a period  of  Time 
until  they  know  enough  about  each  other  that  they  can  work  out  compromises.  While  it 
takes  time,  the  process  is  likely  to  lead  fo  re«m.r.'*fidaffont  uu  IJw  wM  ahd  to 
which  they  can  feel  committed. 

The  issues  to  be  resolved  are  typically  too  complex  and  the  number  of  factors  to  be 
represented  is  typically  too  great  for  a fate-to-late  confrontatior.  to  be  prcrluctiwe.  The 
individuals  who  should  be  involved  in  the  discussion  are  busy  people  who  are  rarely  in  the 
seme  place  et  sarrie  llrne,  ind  the  priority-setting  process  takes  more  time  than  they 
can  comfortably  spare.  For  the  process  to  work,  the  user  representatives  must  perceive 
that  they  have  a great  deal  of  influence  over  the  final  product  without  having  to  become 

» (M.  Turoff,  "The  Design  of  a Policy  Delphi,"  Technological  Forecasting  ^d  Social  Change, 
1970,  2,  149-171) 
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involved  in  the  mechanics  of  the  process.  When  their  advice  is  asked,  they  should  fee 
that  the  questions  asked  are  answerable  and  pertinent  to  the  final  recommendations.  It 
they  feel  important  points  are  being  slighted,  they  should  be  able  to  alert  the  S^’O^P- 
Unlike  technology  forecasting  studies,  it  is  more  important  to  accurately  represent  and 
assimilate  the  opinions  of  group  members  than  to  strive  for  consensus. 

Turoff  (1970)  points  out  that  at  least  three  different  participant  roles  exist  in  this 
type  of  policy  polling  --  the  individuals  seeking  policy  edvice,  the  small  team  that  designs 
questions,  compiles  feedback,  coordinates  the  effort,  and  writes  the  final  report,  and  the 
larger  group  chosen  to  represent  the  various  interests  of  the  user  community.*  In  our 
case,  the  Advanced  Research  Projects  Agency’s  Human  Resources  Research  Office 
requested  a study  of  what  features  should  go  into  innovative  military  CAI/CMI  terminals. 
The  small  group  that  administered  the  study  included  a terminal  designer,  two  experts  on 
computer-assisted  instruction,  and  three  human  communication  researchers  who  acted  as 
intermediaries  between  the  large  and  small  groups.  The  large  group  included  twenty 
experts  or  teams  of  experts  — educational  researchers,  CAI/CMI  developers,  instructional 
technologists,  computer  scientists,  and  CAI/CMI  courseware  authors. 

in  the  following  section,  Ihe  procedures  used  for  eliciting  opinions  during  the  various 
rounds  are  described.  Slightly  different  approaches  were  used  in  each  round.  Questions 
were  derived  from  responses  and/or  unresolved  issues.  We  did  not  presume  that 
statistical  measures  of  significance  could  be  employed  in  analyzing  results.  Instead, 
responses  were  condensed  and  organized  in  a way  that  merged  similar  opinions  while 
preserving  distinctive  ones,  it  is  unlikely  that  this  group  of  participants  would  reach 
conclusions  much  different  from  the  ones  reported  here  if  queried  again.  The  reader  will 
have  to  decide  for  himself  whether  the  participants’  preferences  can  be  generalized  to  the 
entire  CAI/CMI  community. 


♦ Turoff,  oPi  ^ 
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Procedures  and  Materials 


The  basic  procedure  followed  in  this  study  was  an  iterative  query  and  feedback 
technique  using  three  questionnaires  mailed  to  participants.  Other  materials  provided  to 
participants  included  working  papers  and  scenarios,  summaries  of  results  from  each  round 
of  questionnaire  responses,  lists  and  biographical  sketches  of  participants,  and 
miscellaneous  administrative  correspondence.  These  materials  are  contained  in  the 
appendixes.  The  materials  and  specific  procedures  are  described  below. 


Selection  of  Participants  and  Initial  Mailinfi 


Potential  participants  were  selected  by  the  authors  working  in  conjunction  with  Dr. 
William  Mann,  the  principal  investigator  of  the  overall  project  at  the  Information  Sciences 
Institute,  and  ARPA  project  officers,  including  Col.  Austin  Kibler  and  Dr.  Thomas 
O’Sullivan.  Nine  participants  or  participant  teams  were  selected  from  the  military 
community.  Twenty-five  potential  participants  were  selected  from  the  civilian  community 
(business  and  education)  in  the  expectation  that  about  half  would  participate.  It  was 
hoped  that  there  would  be  an  approximate  balance  of  participants  from  the  military  and 
civilian  communities  in  the  final  participant  pool.  This  balance  was  achieved  (see  Table  1). 
Most  of  the  potential  participants  were  contacted  by  telephone  by  one  of  the  authors.  A 
letter  confirming  telephone  responses  and/or  soliciting  participation  in  the  study  was  sent 
to  potential  participants  on  August  30,  1974  (Appendix  II.A).  Enclosed  with  this  letter 
a working  paper  describing  the  study  and  a questionnaire  requesting  biographical 
information  (Appendix  !‘  A).  A list  of  potential  participants  being  contacted  was  also  sent 
in  this  mailing.  Different  cover  letters  and  participant  lists  were  sent  to  potential  civilian 
or  military  participants.  Responses  trickled  in,  and  a few  participants  nominated  other 
individuals  either  in  substitution  for  or  in  addition  to  themselves.  Letters  were  sent  the 
week  of  September  31,  1974,  acknowlecfeing  receipt  of  biographical  questionnaires  (which 
confirmed  the  participant’s  intention  to  participate  in  the  study)  or  requesting  completed 
questionnaires  from  a few  oarticipants  who  responded  by  letter  nnly  . 


A list  of  the  actual  participants  with  notations  regarding  questionnaire  rounds  in 
which  they  participated  is  contained  in  Table  1.  A total  of  25  experts  agreed  to 
participate.  Two  never  responded  further.  The  balance  is  distributed  into  20 

participant-teams,  as  noted  in  Table  1.  Brief  biographical  summaries  of  active  participants 
are  contained  in  Appendix  I.A.  In  addition,  a note  concerf.ing  solicitation  of  military 
participants  is  contained  in  Appendix  I.B. 
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TABLE  1 


Active  Participant  Response  Record 


PARTICIPANT 


Dr.  Ernest  Anadasio 
Dr.  Alfred  BorK 
Dr.  Peter  Dean 
Mr.  Richard  Ditzik 
Dr.  Robert  Fitzhugh  and 
Dr.  Robert  Glaser 
Dr.  John  Ford 
Mr.  Frank  Giunti 
Dr.  Keith  Hall 
Dr.  Albert  Hickey 
CPT  Larry  Hinkle 
Mr.  Donald  Kimberlin 
Dr.  David  Merrill 
Dr.  Leon  Nawrocki 
Dr.  Marty  Rockway 
Dr.  Worth  Scanland 
Dr.  Bruce  Sherwood 
Dr.  Joseph  Ward 
MAJ  D.  A.  Weihe, 

CPT  D.  Glessner,  and 
TSGT  L.  Miller 
Dr.  Jon  Wexler 
Dr.  Karl  Zinn 


ROUNDS 

STATUS 

ONE 

TWO 

THREE 

FOUR** 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

* 

X 

X 

X 

X 

* 

X 

X 

X 

X 

X 

X 

X 

* 

X 

X 

X 

* 

X 

X 

X 

X 

X 

X 

X 

X 

* 

X 

X 

X 

* 

X 

X 

X 

* 

X 

X 

X 

X 

* 

X 

X 

X 

*e  X 

X 

X 

X 

X 

X 

X 

X 

X 

17 

16 

14 

9 

STATUS  KEY: 

^Military 

sParticipated  as  a "team". 


**AII  participants  were  telephoned  in  Round  Four. 
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Questionnaire  Rounds 


Each  round  of  queafiorrrairoa  waa  daaigned  by  Iho  authors.  The  duostionoairo  (or 
each  round  of  query  after  the  first  round  *as  based  upon  the  responses  to  'J® 
round.  Participants  were  sent  an  introduction  to  each  round  and  a summary  P 
to  the  previous  round  for  each  round  after  the  first. 


Questionnaires  and  summaries  are  contained  in  Appendix 


Round  One.  The  questionnaire  for  round  one  was  designed  to  probe  the  6®"®’’“' 
atmude7^r-s;;ce"  of  each  participant  regarding  CAI/CMI,  to  discover  projected  md.lary 
fes  orCAI/Cfdl,  and  to  direct  query  and  feedback  during 

provided  two  or  more  alternatives  and  asked  for  the  responden  s p consensus 

well  as  stances  he  would  be  willing  to  accept  for  purposes  of  consensus.  While  consensus 
rJ  a preprc..  ance  of  preferences  for  one  alternative)  was  achieved  on  some  points 
the’ exercise  was  primarily  useful  in  identifying  and  clarifying  potential  issues  which 
be  raised  m Round  two. 


Along  with  the  questionnaire,  a set  of  scenarios  was  provided  to 
by  suggesting  future  CAI/CMI  environments  in  the  military  (Appendix  II  3). 
exoS  tS  particip.r4  with  no  military  experience  would  use  th  scenarios  as  a 
surrogate  for  Lch  experience,  and  that  participants  with  military  experience  wood 
suggesfrevlns  of  the^cenarios.  Since  no  comments  about  the  scenarios  were  rece.vuo 
from  the  participants,  the  scenarios  were  not  referred  to  again  in  the  course  c.  ...c  study. 
As  shown  in  Table  1,  17  responses  were  received  in  round  one. 


Round  Two.  The  questionnaire  for  round  two  was  developed  after  receiving  the 
mrioritV^reT^nses  (16)  from  round  one.  Round  two  was  used  as  a bridge  between  the 
firit  round  (probing  participants’  stances)  and  the  third  round  (concentrating  on  specific 
terlinaMeatures).  Participants  were  asked  to  respond  to  three  types  of  questions: 


(1)  Anticipated  payoff  of  investment  in  terminal  development; 


(2)  Identification  of  terminal  features  to  be  explored  in  round  three;  and 


(3)  Continued  probing  of  round  one  issues,  such  as  design  of  terminals  for  users 
with  a wide  range  of  motivation  and  intelligence  levels. 

The  questions  were  open-ended  and  succeeded  in  eliciting  substantial 
^ was  ptsibteTo  sens'e  informally  which  issues  were  felt  to  be  signif  cant  and  to  feed  this 
back  to  participants  in  both  the  summary  and  in  the  round  three  questions. 


In  round  two  participants  were  encouraged  to  respond  by  an  o«er  of  a small 
stioend  Civilian  participants  who  responded  did  receive  the  stipend.  Unfortunately,  it 
Jr  later  employed  by  the  military  could  not  be  paid  from 

project  funds;  so,  a letter  informing  them  of  this  problem  was  sent  to  those 
(Appendix  I1.C).  Nevertheless,  7 of  the  9 military  participants  responded  to  round  two,  and 

9 of  the  1 1 civilian  participants  responded. 
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Round  Three.  Following  the  previously  established  procedures,  the  rcund  three 
questionnaire  was  designed  and  sent  to  participants  along  with  a summary  of  round  two 
results  and  a list  of  active  participants.  The  questions  were  structured  so  as  to  get 
degree  of  commitment  information  regarding  particular  policies  and  features.  The  lack  of 
qualifying  remarks  in  the  responses  suggests  that  the  questions  were  appropriate  and  the 
respondents  felt  they  were  ready  to  vote.  ^ 


The  analysis  of  the  third  round  questionnaire  responses  constitutes  the  results 
section  of  this  report.  A total  of  14  responses  were  received  in  round  three. 


Round  Four.  In  a final  mailing  this  report  in  draft  form  was  sent  to  participants  wi;n 
the  request  that  they  communicate  any  comments  and/or  dissenting  opinions  to  the 
authors.  Participants  were  also  contacted  by  telephone.  Dr.  Merrill  said  that  he  thought 
the  emphasis  on  course  development  rather  than  terminal  development  was  consistent  with 
his  feelings.  Dr.  Wexler  stated  that  he  expected  that  the  final  round  would  have  resulted 
in  a more  precise  specification  of  desirable  terminal  features,  with  relative  weights.  Mr. 
Ditzik  said  he  felt  the  study  should  be  replicated  with  a wide"  selection  of  participants. 
Six  other  participants  responded  with  no  further  comments. 
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Results 


The  major  conclusion  of  the  study  is  that  r^ar^  shoujd  ^ 

oedaKOHici-l  software  or  coursewriting  rather  tjian  in  tjie  deyelo^m^  new^Dllj^ 

that  the  terminal  needs  of  both 

combined,  that  investing  in  new  terminal  development  receives  low  prion  V>  _ 

of  terminals  felt  necessary  will  most  likely  be  available  from 

resolution  of  the  stanil-alone/time-sharing  problem  can  be  reached,  and  that 

lel^^es  fett  most  necessary  are  interactive  ones  for  tailoring  mstruchon  to  the 

course-related  needs  of  individual  students. 

1 It  is  in  the  best  interests  of  advancing  the  CAI/CM  terminal  s^^ 

combine  cMfi-hJllte^S^  '-i'  ^ 

needs.  When  asked  this  question,  ten  respondents  favored  combining  tn  V 

zero  favored  keeping  the  needs  distinct,  two  chose  the  category  other  , and  two 
did  not  respond.  The  two  "other"  responses  both  indicated  that  ^ 

militarv  reouirements  and  instructional  situations  requiring  special  term  nals,  but  that 
general^  civilian  needs  can  be  combined.  Almost  every  participant  a some 

Lint  du^ring  the^  study  observed  that  special  terminals  are  needed  in  special  situations. 
Lmetimes  ruggedness  and/or  portability  are  essential.  Other  times  simple  inpu  n 
involving  typing  and/or  verbali  iing  is  important. 


2 In  terms  oi  thejr  potential  payoffs  for  advancing  C^!/CMI  st^l 
(assuming  a~1^985  time  frame),  investment,  in  either  innovative  pedagogical  software 

or  innovative  coursewnting  r^  higher  than  investme^  m 

“ked  to  rank  four  possible  investment  strategies  from  1 (greatest  potential  for  a^vanc  ng 
the  state-of-the-art)  to  4 (least  potential),  terminal  hardware  received  an  average  rating 
of  3 Bo?h  an  investment  in  software  and  an  investment  coursewnting  received  a 
average  rating  of  about  2.  The  fourth  alternative  - an  investment  in  '^rge  scale  use  of 
existing  systeL  - averaged  out  about  equal  to  terminal  hardware.  Exact  tabulation^ 

included  in  Appendix  III.F. 

It  shpuld  not  be  assumed  from  this  ranking  that  an  investment  In  terrninal  hardware 

is  not  considered  important.  In  the  second  f ^ reasons 

investment  in  terminal  development  would  have  a significant  payoff.  Many  of  the  reasons 
giren  m slooort  oj  such  an  investment  argued  that  the  CAl/Cldl  marketplace  was  not  large 
enough  toLtract  special  atfention  from  vendors  and  that  business-oriented  term 
would  no?  meet  hst?uctional  needs.  This  concern  for  the  size  of  the  marketplace  may 
explain  why  it  is  in  the  best  interests  of  civilians  and  military  to  combine  their  markets. 


3.  The  fMiures  M t^  most  n_ece^ 

available  f7^  mainstream  terminal  vendors,.  When  members  were  asked  to  rate  24 
posklTte7;;;rnal  features  in  terms  of  their  necessity  using  the  following  rating  scheme 
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++  definitely  yes 
+ would  be  nice 
0 neutral 
- probably  not 
” definitely  not 

>even  features  received  an  average  rating  between  ++  and  +.  [See  Appendix  III.F, 
question  A(l)  for  details.]  The  two  most  defin‘*c'y  needed  features  (alphanumeric  typed 
input  and  a single  action  to  invoke  frequently  used  functions)  are  availaole  on  most 
existing  terminals.  Only  limited  graphics  features  are  definitely  needed  (an  ability  to  touch 
or  point  to  screen  locations  and  an  ability  to  generate  simple  straight  line  figures). 
Symbol  set  requirements  may  not  currently  be  available  ir  many  terminals,  but  will  not  be 
too  difficult  to  incorporate  (programmable  symbol  sets  and  a variety  of  predefined  symbol 
sets).  The  final  definite  need  is  for  signals  for  controlling  external  equipment,  which  is 
also  not  too  difficult  to  incorporate. 

On  the  other  hand,  the  only  features  with  "neutral"  or  below  ratings  are  precisely 
those  that  might  prove  difficult  to  implement  (spoken  word  input,  shading  and  texture  in 
display,  and  generation  of  large  numbers  of  symbols  in  displays). 

The  remaining  features  fall  generally  in  the  neutral  to  "it  would  be  nice"  range.  At 
the  lower  end  of  the  range  are  color  displays,  memory,  exact  reproductions  of  screen 
images,  computer-composed  speech,  and  lights  under  keys.  At  the  upper  range  are 
signals  received  from  plug-ins,  alphanumeric  printouts,  line  drawing  input,  display  of 
complex  line  drawings,  nigh-  and  low-resolution  stored  visuals,  moving  visuals, 
prerecorded  audio  output,  and  processing  capability. 

It  is  the  considered  opinion  of  the  investigators  that  by  the  time  a terminal  could 
move  from  development  to  large  scale  production,  commercial  vendors  will  be  marketing 
terminals  that  meet  all  but  perhaps  the  least  necessary  of  these  features. 


A.  ^ recommendation  can  be  made  about  whether  terminals  should  be  stand-alone 
CAI/CMI  systems  or  rather  communication  devices  connected  to  time-sharing  systems.  In 
both  the  first  round  and  in  the  third  round,  participants  were  asked  about  whether  or  not 
terminals  should  be  stand-alone  systems.  In  the  first  round  9 ranked  stand-alone  over 
time-shared  with  5 ranking  the  opt’ons  the  other  way.  In  the  third  round  3 felt 
stand-alone  capability  was  definitely  needed,  6 felt  it  would  be  nice,  4 were  neutral,  and  1 
was  slightly  opposed.  While  both  responses  show  a preference  for  stand-alone  systems, 
the  trend  is  not  strong  enough  to  be  called  consensus.  A glance  at  the  comments  in 
Appendix  III.D  will  reveal  why  some  participants  feel  very  strongly  in  favor  of  stand-alone 
systems.  Perhaps  the  best  solution  is  to  rely  upon  the  commercial  marketplace,  which  is 
likely  to  provide  a variety  of  solutions  to  the  stand-alone  problem. 


5.  Turning  t_o  pedagogical  software,  the  features  felt  most  necessary  m CAI/CMI 
software  are  course-related,  student-tailored  and  interactively  oriented.  When  asked  to 
rate  fourteen  possible  software  features  in  terms  of  their  necessity  using  the  following 
rating  scheme. 


I 

! 
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++  definitely  yes 
+ would  be  nice 
0 neutral 

- probably  not 

— definitely  not 

four  features  received  an  average  rating  between  ++  and  +.  [See  Appendix  III.F,  question 
7(1)  for  details.]  The  most  definitely  needed  was  software  permitting  instructional 
sequences  tailored  to  the  abilities  and/or  weaknesses  of  individual  students.  came 

problems  and  examples  responr-ive  to  interests  of  particular  students,  hints  that  reduce 
problem  difficulty,  and  summaries  of  student  progress. 


Immediately  following  these  features  with  average  ratings  of  + or  close  to  + are 
course-related  "intelligent"  features  that  (1)  contrast  concept/strategy  information  with 
Information  about  potential  studcrits,  detect  difficulties,  and  advise  the  instructOTi  (2) 
discover  patterns  of  course-related  behavior  and  advise  the  instructor;  (3)  derive 
course-related  strategies  from  examples  provided  by  the  instructor;  (4)  respond 
mearw'gfuUy  t*  course  related  pfv..bltf(i-sofv4r.g  st  ategity,  (5)  rtspOrid  TireSfiirigfijtly  to 
course  related  questions  or  statements  by  students;  and  (6)  accumulate  course-related 
concepts  and  vocabulary  from  examples  pre  'ided  by  the  instructor. 


Finally  with  ratings  between  + and  neutral  are  features  that  respond  to  (1) 
course-independent  problem-solving  strategies;  (2)  pauses;  (3)  dialogue  cues;  and  (4) 
course-independent  questions. 


While  it  was  not  the  major  intention  of  the  study  to  probe  into  specific  software  or 
courseware  areas  needing  attention,  it  was  felt  from  the  results  of  round  two  that 
exploratory  probing  might  prove  fruitful.  Unlike  the  list  of  terminal  features,  which  was 
assembled  from  round  two  recommendations,  the  software  list  was  assembled  purely  by 
the  investigators.  No  claim  is  made  that  the  best  or  most  relevant  features  were  included. 
Most  likely  a better  analysis  could  be  done  at  some  future  time.  However,  when  asked  for 
additional  software  features,  only  five  of  the  fourteen  respondents  mentioned  anything. 
Two  mentioned  that  they  had  worked  so  long  on  the  problem  that  it  was  hard  to 
summarize  their  opinions.  Two  others  mentioned  the  problem  of  exporting  courseware  for 
use  on  other  systems.  For  more  comments  regarding  authoring  and  adaptable  software, 
see  the  responses  to  question  3 of  round  two  in  Appendix  III.D. 


In  conclusion,  it  would  seem  that  the  major  focus  for  funding  should  center  upon 
software  and/or  courseware  and  not  upon  terminal  hardware.  Perhaps  people  in  the 
computer-based  instruction  field  are  recognizing  that  hardware  cannot  solve  the  problem 
of  good  instruction.  Rockway  summarizes: 


"The  basic  problem  of  so  called  teaching  software  is  that  most  of  the 
’material’  encountered  makes  no  attempt  to  understand  the  student’s 
knowledge  except  as  reflected  by  simple  answers  to  multiple  choice  questions. 
The  material  does  not  carry  out  the  dialogue  of  an  expert  tutor  because  most 
authors  do  not  understand  how  this  is  done  or  because  it  cannot  be 
supported  by  the  computer  available  to  the  author.  ...  We  could  probably 
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profit  both  from  development  of  the  technology  and  technique  of  carrying  out 
verbal  and  manual  dialogue.  The  case  for  an  expensive  electronics 
presentation  system  for  fixed  material  is  very  difficult  to  make." 
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Appendix  LA 

BIOGRAPHICAL  SUMMARIES  OF  PARTICIPANTS 


DR.  ERNEST  J.  ANASTASIO 
Associate  Director 
Data  Analysis  Research  Division 
Educational  Testing  Service 
Princeton,  NJ  08540 
(609)921-9000 


Director  of  Educational  Technology  Research  and  Associate  Director  of 
the  Office  of  Data  Analysis  Research  at  ETS. 


He  has  taught  graduate  studies  in  statistics  and  the  use  of  computers  in 
research  at  Princeton  University  and  the  New  School  for  Social  Resear  m His 
research  interests  are  in  areas  of  instructional  technoiogy,  computing 
methodology  and  the  methodology  of  modern  data  analysis. 


DR.  ALFRED  M.B0RK 
Department  of  Physics 
University  of  California 
Irvine,  CA  92664 
(714)833-6665 


We  have  developed  at  Irvine,  a large  group  of  studorit -computer  dialogs 
for  physics  use,  including  the  underlying  macro-based  software.  Most  of  the 
materiai  uses  graphics,  with  the  Tektronix  4013  or  our  primary  terminal. 
These  materials  are  in  heavy  use  in  our  beginning  classes,  and  are  beginning 
to  see  some  use  on  other  campuses  of  the  University  of  California. 


DR.  PETER  M.  DEAN 
IBM  Corporation 
3424  Wilshire  Blvd. 

Los  Angeles,  CA  90010 
(213)382-7272,  ext.1272 


Ed.D.  Columbia  University  - Teachers  College  Science  Education. 
Manager  Technical  Requirements,  EDEX  teaching  - systems.  Manager 
Interactive  Terminal  Education  Development,  IBM  corporation. 


MR.  RICHARD  DITZIK 
Control  Data  Corp. 


> II 
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BLNISD 

8120  Penn.  Ave.,  S. 
Bloomington,  Minn.  55431 
(612)633-0371,  ext.391 


BSE  (EE),  University  of  Michigan!  M.S.  Cybernetic  Systems,  California 
State  University,  San  Jose.  Presently  representing  Control  Data  Corporation 
(Terminal  System  Division)  efforts  in  developing  an  education  terminal  for 
future  CBE  systems.  Primary  interest  in  computer-base  instructional 
communication  systems. 


DR.  ROBERT  FITZHUGH 
LROC 

University  of  Pittsburgh 
Pittsburgh,  PA  15213 
(412)624-4895 


Computer  scientist  interested  in  system  design  and  educational  use  of 


computers. 


DR.  JOHN  D.  FORD,  JR.  ^ 

Advanced  Instructional  Systems  Directorate 
Naval  Personnel  Research  and  Development  Center 
San  Diego,  CA  92152 
(714)225-7121  or  7140 


Assoc.  Director,  Advanced  Instructional  Systems,  Navy  Personnel 
Research  and  Development  Center,  San  Diego.  Ten  years  experience  at  Navy 
Personnel  and  Training  Research  Lab.,  San  Diego;  SDC,  1958-64;  RAND 
1956-58.  Academic  experience:  Temple  University  and  University  of 
Delaware.  Education;  Ed.D.  Teachers  College,  Columbia  University  1954. 
Research  interests:  Instructional  research  and  technology  development 
Including  CAI/CMI  and  simulation. 


MR.  FRANK  E.  GIUNTI 

Commander,  U.S.  Army  Training  Support  Activity 

Attn:  ATTNG-PA-TS 

Mr.  Frank  Giunti 

Ft.  Eustis,  VA  23604 

(804)878-5801 


Mr.  Frank  E.  Giunti  has  been  serving  as  the  Technical  Director, 
Computerized  Training  Systems  Project,  Product  Manager’s  Office,  since  its 
establishment  in  August  1972.  Prior  to  this  period  of  time  he  served  as  the 
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Chief,  CAI  Division,  US  Army  Signal  Center  and  School  (October  1970  to 
August  1972)  and  initially  as  an  instructional  programmer,  CAI  classroom 
supervisor,  and  CAI  project  planner  (August  1966  to  October  1970). 


CAPT  D.  GLESSNER 
AFMPC/DPMYC 

Randolph  Air  Force  B?se,  TX  78148 
(512)652-2414 


BS  in  business  administration,  State  University  of  New  York;  MBA, 
University  of  Alabama;  8 years  active  duty  with  the  Air  Force.  Teach 
management  at  San  Antonio  College.  AFMPC  representative  on  study  of 
Automatic  Processing  Requirements  of  the  80’s  (SADPR-85)  and  the  Base 
Communications  Mission  Analysis  (BCMA);  developed  and  implemented 
research  method  to  analyze  base  level  functional  requirements. 


DR.  ROBERT  GLASER 
LRDC 

University  of  Pittsburgh 
Pittsburgh,  PA  15213 
(412)624-4895 


Psychologist  interested  in  instructional  research  and  development. 


DR.  KEITH  A.  HALL 

College  of  Education 

The  Pennsylvania  State  University 

201  Chambers  Building 

University  Park,  PA  16802 

<814)865-0471 


Graduate  study  in  instructional  systeno  and  technology  and  educational 
psychology  - Indiena  University.  Research  interestes  in  adaptive,  interactive 
instructional  systems.  Management  responsibilities  for  4 CAI  systems  - 1 
fixed  site  and  3 mobile  systems. 


DR.  ALBERT  E.  HICKEY 
Entelek  Incorporated 
42  Pleasant  St. 
Newburyport,  MA  09150 
(617)465-3000 
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/resident  of  Entelek,  Inc.  Consultant  to  ETS  ai^d  ARPA.  Author  of 
Research  Guidelines  for  C^.  Background  in  experimental  psychology  and 
human  engineering.  Author  of  PI  & CAI  programs,  especially  for  industry. 


CPT  LARRY  HINKLE 

Commander,  U.S.  Army  Training  Support  Activity 

Attn:  ATTNG-PA-TS 

CPT  L Hinkle 

Ft.  Eustis,  VA  23604 

(804)878-5801 


No  biographical  information  received. 


MR.  DONALD  A.  KIMBERLIN 
CTS  Field  Office 
TRADOC 
Project  ABACUS 
Signal  Towers,  Room  709 
Ft.  Gordon,  GA  30905 
(404)791-3193  or  7297 

Mr.  Donald  A.  Kimberlin  served  as  an  instructional  programmer, 
course  development  team  chief,  and  classroom  supervisor  in  the  CAI  Division 
from  1968  to  1972.  From  1972  to  the  present,  Mr.  Kimberlin  has  served  as 
an  Educational  Specialist  and  Chief  of  Course  Development  for  the,  CTS 
Project.  He  is  now  the  Chief  of  the  Course  Development  and  Applications 
Division,  Project  ABACUS. 


DR.  DAVID  MERRILL 
Department  of  Education 
Brigham  Young  University 
Provo,  UT  84601 
(801)224-2350 

My  Ph.D.  was  obtained  from  the  University  of  Illinois  under  Larry 
Stolurow,  my  dissertation  being  one  of  the  first  studies  on  SOCRATES.  I have 
published  in  the  area  of  instructional  design.  I was  leader  of  the  team  which 
did  courseware  design  for  the  TICCIT  system.  This  design  was  based  largely 
on  the  theoretical  work  which  I had  done  on  instructional  design.  I am 
currently  on  sabbatical  leave  from  B.Y.U.  and  serving  as  Vice-President  of 
Courseware,  Inc.  which  is  currently  involved  in  two  major  projects  training 
military  personnel  to  develop  CAI  materials. 
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TSGi'  A.  MILLER 
AFMPC/DPMYC 

Randolph  Air  Force  Base,  TX  78148 
(512)652-2414 


Entered  the  Air  Force  in  1955;  participated  in  the  tests  the  Air  Force 
was  conducting  prior  to  release  of  CAI  Air  Force  wide  in  1972;  has  directed 
training  programs  and  managed  CAI/CKil  systems. 


DR.  LEON  H.  NAWROCKI 
Computer  Instruction  Research  Program 
U.S.  Army  Research  Institute 
Commonwealth  Building 
1300  Wilson  Blvd. 

Arlington,  VA  22209 
(202)694-3954 

Dr.  Nawrocki  received  his  Ph.D.  from  the  Ohio  State  University  in 
1969  and  has  since  been  employed  by  the  Army  Research  Institute.  From 
1969  to  1972  he  was  assigned  to  Command  Systems  unit  and  conducted 
researcn  on  information  displays.  From  1972  to  present  he  has  been  senior 
psychologist  in  the  Educational  Technology  unit.  Organizational  membership 
includes  APA  (Divs.  1 & 21),  HFS,  ADCIS  and  AERA. 


DR.  MARTY  R.  ROCKWAY 

Technical  Training  division 

Air  Force  Human  Resources  Laboratory 

Lowry  A’-  Force  Base,  CO  80230 

(303)394- 1385 


L'arty  R.  Rockway  is  a native  of  Chicago,  Illinois.  After  completion  of 
undergraduate  work  in  the  physical  and  engineering  sciences  he  received  a 
Ph.D.  in  psychology  and  statistics  from  Northwestern  University  in  1953.  In 
1963-64  he  was  a Princeton  Fellow  at  the  Woodrow  Wilson  School  of 
Princeton  University  where  he  majored  in  the  areas  of  public  administration 
and  national  security  affairs.  During  1967-69  he  was  a Littauor  Fellow 
engagea  in  a joint  program  in  management  science  and  science  and  public 
policy  at  Harvard  and  M.I.T. 

During  the  past  twenty  years  Dr.  Rockway  has  held  a number  of  R and 
D posts  within  the  Air  Force  Systems  Command,  including  the  position  of  Chief 
Engineer  for  Human  Factors  at  the  Aeronautical  Systems  Division  and  his 
current  position  as  Technical  Director,  Technical  Training  Division,  Air  Force 
Human  Resources  Laboratory. 
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DR.  WORTH  SCANLAND 

Chief  of  Naval  Education  and  Training 

Code  N-330 

Nava!  Air  Station  Pensacola,  Fla.  32508 
(904)452-3466 


Attended  the  Naval  Academy  followed  by  30  years  active  duty  as  a 
Naval  officer,  mostly  submarines,  followed  by  graduate  studies  at  FSU,  with  a 
M.S.  in  educational  research  and  a Ph.D.  in  Instructional  Technology- 
followed  by  a couple  of  years  as  director  of  research  with  the  Florida  Youth 
Services  Authority  (juvenile  delinquency),  followed  by  current  duties  wtth  the 
Naval  Education  and  Training  Command  staff. 


DR.  BRUCE  SHERWOOD 
Room  252 

Engineering  Research  Lab 
University  of  Illinois 
Urbana-Champaign,  IL  61801 
(217)333-6210 


B.S.  Engineering  Science,  Purdue  Univ.-1960 
Fulbright  to  Padova,  Italy 
Ph.D.  Physics,  University  of  Chicago  - 1967 


Taught  and  did  experimental  particle  physics  research  at  Caltech 
1966-1969.  At  University  of  Illinois  (Urbana)  since  1969  - now  Assistant 
Director  of  the  Computer  based  Education  Research  Laboratory  (PLATO)  and 
Associate  Professor  of  Physics.  Worked  on  design  and  implementation  of 
TUTOR  language.  Developed  PLATO  version  of  introductory  classical 
mechanics  course.  Author  of  the  text  The  TUTOR  Language. 


DR.  JOSEPH  S.  WARD 
U.  S.  Army  Research  Institute 
Commonwealth  Building  - Room  2045 
1300  Wilson  Blvd. 

Arlington,  VA  22209 
(202)694-1397 


Dr.  Ward  has  worked  in  the  design,  development,  management,  and 
evaluation  of  CAI/CMI  systems  in  Army  training  programs.  His  primary 
interests  in  this  study  are  in  instructional  systems  development  research 
involving  CAI/CMI  as  one  delivery  mode  of  instruction. 
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MAJ  D.  A.  WEIHE 
AFMPC/DPMYC 

Randolph  Air  Force  Base,  TX  78148 
(512)652-2414 


I have  a degree  in  secondary  education  from  the  University  of  Wichita 
(now  Wichita  State  University)  and  16  years  experience  in  Air  Force 
personnel  management.  I have  27  hours  of  graduate  study  leading  to  an 
MS  in  systems  management  trom  St.  Mary’s  University,  San  Antonio,  Texas. 


DR.  JON  WEXLER 
(During  study) 

Department  of  Computer  Science 
State  University  of  New  York  at  Buffalo 
4226  Ridge  Lea  Rd. 

Amherst,  NY  14226 
(Currently  at  Tempe,  AZ) 
(602)967-3248 


Research/teaching  interests  are  in  the  area  of  artificial  intelligence  and 
the  application  of  its  representations  and  processes  to  computer-based 
teaching  systems  to  generate  portions  of  the  material  needed  for  intelligent 
(interesting)  student-computer  dialogue.  Involved  in  the  (slow)  development 
of  a generative  teaching  system  for  multiple  programming  languages;  current 
work  focuses  on  the  generation  of  equivalent  target  language  programs  from 
a visually-oriented  ab'tract  language. 


DR.  KARL  L ZINN 
Research  Scientist 
University  of  Michigan 

Center  for  Research  on  Learning  and  Teaching 
109  E.  Madison  Street 
Ann  Arbor,  Ml  48104 
(313)763-4410  or  0158 

Karl  L.  Zinn,  Research  Scientist  at  the  Center  for  Research  on  Learning 
and  Teaching,  and  Associate  Director  of  the  MERIT  Computer  Network  at  the 
University  of  Michigan,  is  engaged  in  development  of  innovative  uses  of 
computers  in  education,  giving  special  attention  to  computer  languages  and 
supportive  systems.  He  has  worked  with  dozens  of  curriculum  authors  in  a 
variety  of  subject  areas  who  have  prepared  learning  exercises  usin^  ten 
different  authoring  languages  as  well  as  a number  of  general-purpose 
programming  languages. 
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Appendix 

MILITARY  PARTICIPANT  SELECTION 


MEMORANDUM  FOR  THE  RECORD 


From:  Monty  C.  Stanford 

Date:  October  28,  1974 

Subject:  Obtaining  Military  Participants  in  the  CAI  Study 


Military  participants  were  obtained  for  the  CAI  terminal  study  in  the  same  manner  as 
civilian  participants.  That  is,  the  same  kind  of  letter  requesting  participation  in  the  study 
was  sent  to  military  participants  as  was  sent  to  civilian  participants.  Names  of  military 
participants  were  obtained  from  recommendations  and  by  contacting  key  agencies  in  the 
various  armed  services. 


One  reason  for  using  this  method  was  to  attempt  to  ensure  that  a participant  from  one  of 
the  armed  services  was  not  participating  merely  because  he  or  she  had  been  assigned  by 
his  or  her  superior  to  participate.  I still  feel  that  such  "volunteer"  participation  is  good 
and  is  a desirable  aspect  of  tnis  and  future  studies. 


However,  several  military  personnei  expressed  some  hesitancy  to  participate  in  the  study. 
They  seemed  to  be  unsure  as  to  whether  or  not  such  participation  would  be  sanctioned, 
approved,  or  required  by  higher  command.  And,  in  one  case,  the  potential  participant  we 
contacted  desired  to  participate  in  the  study,  but,  after  receiving  our  initial  materials,  the 
person’s  superiors  denied  permission  to  participate  in  the  study. 


Recommendation: 


It  is,  therefore,  recommended  that  in  future  studies  of  this  type  participation  be  requested 
from  military  personnel  on  a volunteer  basis  in  the  same  manner  as  was  done  in  this  study. 
But,  it  is  also  recommended  that  the  highest  command  level  possible  be  contacted  and 
approval  for  participation  in  the  study  be  obtained  before  requests  are  made  to  individual 
military  personnel. 
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Appendix  II.A.l.a 

COVER  LETTER  TO  CIVILIAN  PARTICIPANTS 


August  29,  1974 


[Name  of  participant]  [Address  of  participant] 
Dear  [Title  and  last  name]: 


The  Information  Sciences  Institute  and  the  Annenberg  School  of  Commui.ications  at 
the  University  of  Southern  California  are  conducting  a study  for  the  Advanced  Research 
Projects  Agency  in  order  to  learn  from  experts  what  features  should  be  included  in 
pace-setting  CAI  terminals  five  to  ten  years  from  now  and  why.  The  intention  of  the 
study  is  to  discover  if  there  are  new  devices  and/or  strategies  for  making  CAI  user 
interfaces  more  effective. 

Would  you  be  willing  to  participate  on  a panel  of  experts  from  October  1974  until 
January  1975?  We  expect  to  conduct  four  rounds  of  query  and  feedback  regarding 
possible  features  and  reasons  for  the  features.  Enclosed  is  a working  paper  detailing  the 
goals  of  the  study  and  procedures  we  plan  to  follow,  a questionnaire  that  will  help  us  all 
better  understand  each  other,  and  a list  of  the  other  people  who  are  being  contacted. 
Please  let  us  know  as  soon  as  possible  whether  or  not  you  can  participate  and  who  else 
we  ought  to  contact. 


Sincerely  yours, 

/s/Monty  Stanford 

for/  Bill  Mann,  ISI 

Rick  Carlson,  ASC 
Tom  Martin,  ASC 
Monty  Stanford,  ASC 


Enclosures 
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Appendix  II.A.l.b 

COVER  LETTER  TO  MILITARY  PARTICIPANTS 


[Name  of  participant] 
[Address  of  participant] 


Dear  [Title  and  last  name]; 


In  reference  to  our  recent  telephone  conversation,  I am  enclosing  materials  on  the 
Computer  Assisted  Instruction  Terminal  Study  being  conducted  by  the  Information  Sciences 
Institute  and  the  Annenberg  School  of  Communications  at  the  University  of  Southern 
California  for  the  Advanced  Research  Projects  Agency.  We  hope  to  learn  from  experts 
what  features  should  be  included  in  pace-setting  CAI  terminals  five  to  ten  years  from  now 
and  why.  The  intention  of  the  study  is  to  discover  if  there  are  new  devices  and/or 
strategies  for  making  CAI  user  interfaces  more  effective. 


Through  the  course  of  the  study,  we  expect  to  conduct  four  rounds  of  query  and 
feedback  regarding  possible  features  and  reasons  for  the  features.  Enclosed  is  a working 
paper  detailing  the  goals  of  the  study  and  procedures  we  plan  to  follow,  a questionnaire 
that  will  help  us  all  better  understand  each  other,  and  a list  of  the  other  people  in  the 
military  community  who  are  being  contacted.  We  have  also  enclosed  brief  vitas  of  the 
ASC  study  members,  so  that  you  will  have  some  idea  of  who  we  are. 


In  addition  to  participants  from  the  military  community,  we  are  contacting  potential 
participants  from  academic  and  business  organizations  who  are  CAI  users  or  researchers. 
Brief  biographical  statements  on  all  participants  will  be  sent  to  you  with  the  first  round  of 
questionnaire  materials. 

We  look  forward  to  your  participation  in  this  study.  Please  let  us  hear  from  you 
as  soon  as  possible. 


Sincerely  yours, 


/s/Monty  Stanford 


for/  Bill  Mann,  ISI 

Rick  Carlson,  ASC 
Tom  Martin,  ASC 
Monty  Stanford,  ASC 


Enclosures 


m 
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Appendix  H.A.2 
WORKING  PAPER 


The  ARPA  Contract 

There  are  a number  of  tasks  included  in  the  contract  between  the  Information 
Sciences  Institute  and  ARPA’s  Human  Resources  Research  Office,  only  one  of  which  is  the 
querying  of  experts.  Some  of  the  other  projects  that  will  be  completed  during  the  first 
year  include  1)  putting  PLATO  terminals  on  the  ARPA  network,  2)  surveying  the  literature 
for  CAI  user  interface  descriptions,  3)  tracking  technological  developments  that  might 
significantly  advance  CAI  terminal  state-of-the-art,  4)  transforming  the  recommendation  of 
the  panel  of  experts  into  specifications  for  a terminal  that  can  be  sent  out  to  contractors. 
The  major  long  range  study  is  directed  toward  discovering  and  modeling  human  discourse 
processes  that  can  later  be  incorporated  into  interactive  computer  systems. 

Goals  for  the  Querying  of  Experts  Task 

Available  terminals  frequently  limit  what  authors,  designers,  and  researchers  can  do 
with  CAI  or  CMI  systems.  In  order  to  have  equipment  on  hand  five  years  from  now  that 
meets  needs  then,  we  must  find  out  what  designers,  teachers,  and  researchers  plan  to  be 
doing  and  what  they  will  need  in  a terminal.  While  we  must  produce  an  end  product  that 
can  be  turned  into  specifications,  we  will  not  limit  ourselves  to  terminals.  If  it  is  more 
important  that  certain  types  of  software  be  developed,  or  that  new  types  of  learning 
laboratories  be  established,  we  wtnt  to  find  that  out.  We  are  particularly  interested  in 
what  terminal  features  that  are  not  normally  needed  are  needed  for  carrying  out  user 
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interaction  research.  We  suspect  many  of  you  have  good  ideas  about  data  capturing 
techniques,  monitoring  devices,  and  plug-in  features  you  feel  you  need  but  which  cannot 
be  justified  in  mass-produced  terminals.  However,  if  discussion  becomes  too  blue  sky,  we 
plan  to  move  back  toward  operational  environments.  We  are  expected  to  make 
recommendations  regardimj  stand-alone  versus  networked  systems,  but  will  fight  hard  to 
keep  this  from  becoming  the  sole  topic  of  discussion. 

It  is  very  important  that  we  not  come  up  with  a list  of  features  without  filling  in  the 
reasoning  behind  the  features.  We  want  to  know  what  payoffs  you  see  in  the  features 
you  recommend.  While  it  may  not  be  possible  to  justify  features  in  a strict  cost/benefit 
sense,  curiosity  alone  is  probably  not  enough.  We  hope  you  use  the  justification  process 
as  an  opportunity  for  influencing  each  other. 

One  final  note:  ARPA  is  interested  in  helping  to  make  military  education  more 
efficient  and  effective.  While  the  recommendations  we  come  up  with  are  likely  to  have 
wide  applicability,  they  must  take  into  consideration  the  type  of  student,  teacher,  and 
learning  environment  encountered  in  the  military.  We  plan  to  provide  background  material 
to  those  of  you  who  are  unfamiliar  with  Armed  Services  education. 


Procedures  We  Plan  to  Follow 

In  many  respects  we  are  planning  to  have  the  study  be  like  a Delphi  study.  There 
will  be  a series  of  rounds,  feedback  will  be  used  to  bring  participants  towards  consensus, 
names  will  not  be  mentioned,  and  most  of  the  interaction  will  be  through  the  mail. 
However,  we  are  not  interested  in  finding  out  when  you  think  some  development  will  take 
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place,  but  rather  what  you  want  to  have  done.  In  the  early  stages;  we  plan  to  be  frankly 
nonstatistical.  We  are  more  interested  in  letting  you  inform  each  other  about  your 
positions  than  in  forcing  you  to  react  to  ours.  In  order  to  help  you  we  plan  to  provide 
you  with  background  scenarios,  sample  position  statements,  and  checklists  of  things  abuut 
which  others  might  want  to  know  your  opinions.  In  the  first  few  rounds  we  are  really 
looking  for  statements  that  will  bolster  the  final  selection  of  features. 

The  final  rounds  are  intended  to  be  much  more  concrete  and  feature-oriented.  You 
will  be  asked  to  rate  how  important  you  feel  various  features  to  be,  and  to  agree  with  or 
attempt  to  reword  supporting  reasons.  We  expect  that  some  of  you  will  feel  the  need  to 
talk  things  over  with  us  or  with  each  other.  We  plan  to  be  available  via  the  telephone, 
can  make  a limited  number  of  site  visits  (provided  there  are  clear  and  compelling  reasons), 
hope  to  use  teleconferencing  via  the  ARPA  network  and/or  the  PLATO  network,  and  may 
need  to  hold  a one-day  workshop  in  order  to  reach  consensus.  We  welcome  suggestions 
from  you  regarding  how  to  improve  upon  the  querying  procedure. 

The  final  procedural  issue  is  one  for  which  we  have  no  good  answer— how  to  come 
up  with  a single  set  of  recommendations.  A number  of  participants  are  bound  to  feel 
there  is  a need  for  a number  of  different  types  of  terminals.  Other  participants  will  feel 
that  features  cannot  be  talked  about  in  isolation.  We  may  be  forced  to  come  up  with  a 
family  of  terminals  each  responding  to  a different  environment,  or  a family  ranging  from 
cheap  to  expen.;ive.  Features  are  likely  to  cluster  into  groups,  with  some  groups  less 
essential  than  others,  and  some  groups  mutually  exclusive.  The  match  between  features 
and  reasons  for  features  is  not  likely  to  be  one-to-one.  However,  we  will  do  our  best  to 
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keep  participants  working  toward  a loosely  ranked  list  of  justified  features  for  a single 
terminal. 

If  you  have  any  further  questions,  call  Rick  Carlson,  Tom  Martin  or  Monty  Stanford 
at  (213)  7A6-6273.  We  want  you  to  feel  that  the  study  is  for  your  benefit  more  than  for 
ours,  and  hope  that  you  will  let  us  know  how  it  can  be  made  more  responsive  to  your 
needs.  We  think  the  procedure  will  work  and  that  the  potential  payoffs  are  great  enough 
to  make  it  a worthwhile  endeavor  for  everyone  involved. 
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Appendix  II.A.3 


BIOGRAPHICAL  QUESTIONNAIRE 

In  order  to  give  all  of  us  who  are  participating  in  this  study  some  idea  of  the 
composition  of  the  participant  group,  we  would  appreciate  your  completing  and  returning 
to  us  this  biographical  questionnaire. 

If  you  have,  in  addition,  a one-page  vita  or  references  to  papers  or  articles  that  you 
feel  reflect  your  current  thinking  on  the  topic  of  this  study,  it  might  be  helpful  for  you  to 
include  these  also. 


1.  Your  experience  relative  to  computer  assisted  or  computer 
managed  instruction; 

1.1  In  general  it  would  be  helpful  to  know  what  experience 
you  have  had  working  with  CAI/CMI  systems  that  you 
feel  is  relevant  to  the  topic  of  this  study.  But 
first,  there  are  some  specific  items  of  information 
that  we  would  like  to  obtain  from  all  participants. 

For  these  items,  would  you  please  check  the  appro- 
priate items  below.  For  each  item,  circle  CAI,  CMI, 
or  both. 

( ) I have  taken  courses  using  CAI/CMI  systems. 

( ) I have  authored  CAi/CMI  coursework. 

( ) I have  designed  and/or  programmed  CAI/CMI  programs. 
( ) I have  managed  a CAI/CMI  system. 

( ) I have  designed  terminal  hardware. 

( ) I have  other  relevant  experience,  including: 


1.2  Now,  could  you  describe  your  working  experience  that 
you  feel  is  relevant  to  the  topic  of  this  study? 
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2.  Your  experience  relevant  to  military  training: 

2.1  Again,  since  the  ultimate  goal  of  this  study  is  to 

provide  recommendations  for  CAI  systems  in  the  military 
training  environment,  it  would  be  helpful  to  know  what 
kind  of  familiarity  you  have  with  that  environment. 

( ) I have  been  a student  in  a military  service 
training  course. 

( ) I have  taught  in  a military  service  training  program. 
( ) I have  authored  military  training  materials. 

( ) I have  directed  a training  or  education  program 
at  the  local  command  level  or  higher. 

( ) I have  worked  on  the  staff  of  a military  service 
training  command  (or  at  DOD  level). 

( ) I have  other  experience  with  military  training, 
including: 


2.2  Now,  could  you  dc sci  >be  your  working  experience  with 
military  training  that  you  feel  is  relevant  to  the 
topic  of  this  study? 


3.  Your  availability  and  access  to  computer  networks: 

3.1  Are  there  any  times  during  the  conduct  of  this  study 
when  you  wili  not  be  available? 


i 
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3.2  If  you  have  access  to  the  PLATO  network  or  to  the 
Advanced  Research  Projects  Agency  (ARPA)  network, 
it  might  be  most  convenient  to  communicate  with  you 
via  one  of  the  networks.  Do  you  have  access  to 
either  or  both  of  these  networks? 

( ) PLATO 
( ) ARPA 


3.3  If  you  have  access  to  either  or  both  of  these  networks, 
would  prefer  to  participate  in  the  study  via  one 
or  the  other? 


A.  Your  suggestions  and  comments  on  this  study: 


A.l 


What  benefits  do  you  anticipate  or  would  you  like  to 
derive  from  participating  in  this  study? 
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Do  you  have  any  comments  on  the  des 
for  this  study  and/or  do  you  have  any 
the  study? 
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5ndix  II.B.1 


INTRODUCTION  TO  ROUND  ONE 


As  you  know  from  reading  the  working  paper  (distributed  at  the  beginning  of  the 
study),  the  end  product  of  the  querying  process  must  be  translatable  into  specifications 
for  future  CAI/CMI  terminals.  In  this  round  we  are  hoping  to  find  out  what  some  of  your 
general  attitudes  are,  how  you  see  CAI/CMI  fitting  into  the  military  environment,  and  in 
what  areas  we  can  direct  discussion  during  future  rounds.  Enclosed  you  will  find  a set  of 
nine  scenarios  and  a questionnaire.  The  scenarios  are  intended  to  orient  participan  by 
suggesting  (not  delimiting)  how  CAI/CMI  terminals  might  be  used  in  1980-1985.  They 
hopefully  will  act  as  a surrogate  for  military  CAI/CMI  terminal  projected  usage  data.  The 
questionnaire  contains  eight  general  attitude  questions  and  fifteen  usage  environment 
questions.  We  do  not  view  the  questionnaire  as  a validated  instrument  for  gathering 
reliable  data.  Rather  we  view  it  as  a sounding  board  for  stimulating  and  focusing 
discussion.  Notice  the  wide  empty  margins  next  to  questions.  We  want  to  know  what  the 
questions  mean  to  you  and  how  you  would  follow  up  on  them  in  succeeding  rounds,  ypti 
h^  no  attitudes  about  an  issue.  ^ not  feej  compelled  to  place  a i V next  to  one  of  the 
alternatives.  If  your  attitude  has  stipulations,  tell  us  what  those  stipulations  are. 

Some  participants  have  asked  why  we  are  diverging  from  the  Delphi  methodology 
and  whether  or  not  we  view  our  methodology  as  reliable.  From  a social  scientific  view, 
the  querying  process  we  are  employing  cannot  be  called  reliable  — the  sample  size  is  too 
small,  we  are  querying  experts  rather  than  end  users,  the  participants  were  not  chosen  at 
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random  from  an  homogeneous  population,  and  the  questions  have  not  been  thoroughly 
pre-tested.  Nevertheless,  we  believe  tha*.  it  is  better  to  ask  your  opinions  than  to 
pretend  we  already  know  all  we  need  to.  Querying  panels  of  experts  is  a common 
practice  when  esvablishing  standards  for  hard  and  soft  technologies.  If  we  all  keep  in 
mind  the  limitations  of  the  methodology  and  the  vastly  differing  backgrounds  we  come 
from,  perhaps  we  can  educate  each  other. 

Delphi  studies  have  come  under  criticism  recently  in  a report  by  Harold  Sackman  of 
the  Rand  Corporation  (Delphi  Assessment:  Expert  Opinion,  Forecasting,  and  Grou&  Process, 

M.  Sackman,  The  Rand  Corporation  R-1283-PR,  April  1974,  117  pp.).  We  are  attempting 
to  obviate  some  of  his  objections  to  Delphi.  He  objects  to  the  estimating  of  future  dates, 
which  we  are  not  attempting  to  do.  He  argues  that  the  anonymity  of  participants  leads  to 
a lack  of  responsibility  - we  plan  to  summarize  where  possible,  but  not  to  guarantee 
anonymity.  He  objects  to  lack  of  supportive  reasoning  behind  predictions,  so  we  plan  to 
ferret  out  the  reasoning  of  each  participant  regarding  the  more  significant  questions. 
Vv'hile  we  are  hoping  for  consensus,  we  do  not  plan  to  punish  outliers,  and  instead  plan  to 
bring  their  arguments  to  the  attention  of  all  so  that  the  most  persuasive  reasoning  can 

prevail. 

Feel  free  to  challenge  us  on  the  methodology.  If  you  think  the  profiles  are  a waste 
of  time,  tell  us.  If  you  think  they  could  be  revised  to  really  get  at  the  essence  of  military 
insVuction,  revise  them  or  make  up  your  own.  If  you  think  questions  are  unclear, 
ambiguous,  or  too  "lumpy",  please  suggest  revisions.  The  second  round  will  cover  terminal 
input /output  features,  but  the  third  round  will  return  to  the  topics  covered  in  round  one. 
It  will  be  as  exciting  as  the  responses  you  are  about  to  send  us  can  make  it. 
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SCENARIOS 


Seaman  Jones  checks  into  his  local  CAIFAC  (Computer  Assisted  Instruction  Facility) 
for  a lesson  on  the  Mark  IV.  Mod  2,  Radar  Repeater.  Jones  is  a striker  for  ET3  (i.e  he  is 
trying  to  make  the  rating  of  Electronics  Technician  Third  Class)  and  he  is  preparing  for  the 

ET3  Rating  Exam. 


After  showing  his  pass  to  the  guard  to  ensure  that  he  has  the  required  security 
clearance,  Jones  goes  to  the  nearest  available  terminal  and  logs  into  the  ET3  course.  He 
selects  lesson  21  from  the  menu.  The  introduction  to  the  lesson  informs  him  that  he  wil 
need  a mockup  of  the  radar  repeater  for  this  lesson.  Jone:.  goes  to  the  instructional 
equipment  area,  finds  the  mockup  on  a rolling  cart,  and  takes  it  back  to  the  terminal. 
Jones  then  proceeds  with  the  lesson. 


Journalist  Second  Class  (J02)  Bill  Brown  is  studying  for  the  advanced  rating  exam, 
the  JOl/JOC  exam.  He  has  logged  onto  the  CAI  system  aboard  the  USS  CONSThLLATIO 
(CVA-64),  an  aircraft  carrier  currently  on  station  in  the  Western  Pacific. 


J02  Brown  is  currently  reviewing  newspaper  editing  procedures  and  has  been 
presented  with  a diagnostic  test  on  photo  layout.  This  test  presents  him  with  a story  t, tie 
and  a number  of  photographs.  Brown’s  job' is  to  indicate  the  way  he  would  crop  (cut  and 
trim)  the  photos,  the  captions  or  cutlines  he  would  write  for  each  photo,  and  how  he  would 
arrange  the  photos  and  copy  blocks  (captions  or  cutlines)  on  a standard  sized  newspaper 

page. 


Brown  will  be  using  both  the  light-pen  to  indicate  cropping  and  layout  and  the 
keyboard  for  entering  caption  text.  This  lesson  is  the  sixteenth  in  *he  preparatory  course 
for  the  JOl/JOC  Exam  and  Brown  has  also  completed  the  J03/J02  course  (35  lessons)  as 
well  as  a special  short  course  on  photojournalism  (10  lessons). 


THREE: 


The  Education  Services  Officer  at  Clark  AFB,  Phiiipfjines,  is  reviewing  the  training 
records  of  the  men  at  his  command.  From  this  review  he  must  determine  w^hat 
advancement  examinations  to  requisition  from  Training  Command  Headquarters.  This 
review  is  relatively  simple,  since  most  of  the  courses  have  been  CAI  and  records  have 
been  automatically  maintained.  The  ESO  can  obtain  a printout  of  these  records  in  a 
variety  of  formats  by  using  an  author  level  program  which  he  learned  in  a CM  course. 


i"  r--- 
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After  reviewing  the  records,  the  ESO  can  also  have  notices  for  all  the  men 
automatically  prepared  informing  them  that  their  rating  exam  has  been  ordered.  Ip  any 
case  the  CAI  system  will  automatically  record  a notice  for  each  student  informing  him  thai 
his  records  have  been  reviewed,  which  records  were  reviewed,  when,  and  by  whom. 


FOUR: 


Lieutenant  Williams  is  updating  a CAI  course  on  Japan  for  personnel  who  are  being 
transferred  to  Japan.  In  this  course  the  student  is  introduced  to  the  history,  culture,  and 
customs  of  the  Japanese.  The  student  is  also  given  information  specifically  relevant  to  the 
conduct  of  American  armed  forces  personnel  in  Japan.  This  information  ranges  from 
status  of  forces  agreement  information  to  sensitive  topics  such  as  nuclear  power  and 
weapons. 

Several  thousand  personnel  ranging  from  recruits  just  out  of  basic  training  to  field 
grade  officers  (up  to  Colonel)  take  this  course  or  particular  lessons  from  it  each  year  and, 
it  is  available  at  all  commands  and  installations  throughout  the  world.  Some  of  the 
information  changes  frequently  and  the  course  has  to  be  updated  semi-annually.  Once 
Lieutenant  Williams  finishes'  updating  the  course  materials  in  the  courseware  at  the 
origination  point,  all  future  students  receive  the  updated  version  of  the  course. 


FIVE: 


Airman  Farrel  has  just  reported,  along  with  the  other  new  men  in  his  squadron,  to 
the  Base  Firefighting  School.  After  checking  into  the  school,  the  men  are  sent  to  the  CAI 
Terminal  Room.  A sergeant  there  gives  the  men  a half-hour  lecture  with  platform 
demonstration  on  the  use  of  the  CAI  terminal.  The  men  then  go  to  individual  carrells 
where  they  begin  the  first  of  three  half-hour  lessons  on  firefighting.  They  will  receive 
lesson  two  day  after  tomorrow  and  lesson  three  two  days  after  that.  Few  if  any  of  the 
men  have  ever  seen  a CAI  terminal,  so  the  sergeant  and  several  of  his  assistants  wander 
about  the  room  providing  individual  assistance. 


SIX: 


Technical  Sergeant  Maxwell  Denver  had  just  begun  the  third  phase  of  his  training  on 
the  ARC  23  Mod  6 Mark  XV  SSB  FSK  Teletype.  He  had  been  working  on  the  ARC  23 
frequency  standard  and  is  now  about  to  start  the  trouble  shooting  procedures  for  the 
emitter  follower  in  the  first  stage  of  the  demodulator. 

He  begins  by  taking  the  pretest  module,  and  being  assigned  a PC  board  mockup 
which  he  plugs  into  his  test  stana.  (Two  modules  earlier  he  learned  the  peculiarities  of 
the  extenders  and  their  test  points,  and  he  is  using  the  appropriate  extender  now.) 
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Denver  will  be  told  as  he  probes  the  circuit  under  test  the  appropriateness  of  the 
points  he  selects  for  viewing  on  his  simulated  oscilloscope.  If  he  becomes  lost  in  the  hunt 
for  the  malfunction,  he  will  be  prompted  ?nd  perhaps  receive  a short  review  of  the  basic 
principles  he  is  using  in  solving  the  problem. 


SEVEN: 


Staff  Sergeant  James  Kildaire  is  taking  a special  CAI  course  on  emergency 
diagnostics  as  part  of  his  paramedical  training.  He  is  proceeding  through  lesson  14  of  the 
30  lesson  course.  In  this  lesson  he  is  given  a list  of  symptoms  for  a patient  in  a combat 
situation  which  is  also  described.  Sergeant  Kildaire  may  ask  for  additional  information 
about  the  patient  which  he  would  normally  be  able  to  obtain  under  the  given  conditions. 
Sergeant  Kildaire  must  then  enter  his  diagnosis  of  the  patient’s  condition  using  standard 
medical  terminology  and  suggest  emergency  treatment  procedures. 


In  previous  lessons  he  has  then  received  advice  from  a medical  officer  with  whom  he 
was  in  radio  contact.  In  this  lesson,  however,  he  is  on  his  own.  The  CAI  program  accepts 
the  Sergeant’s  treatment  procedures  and  then  informs  the  Sergeant  of  the  effects  on  the 
patient.  The  Sergeant  then  recommends  further  treatment  or  calls  in  a medevac  team. 
This  dialogue  is  continued  until  the  patient  is  evacuated  or  expires. 


EIGHT: 


Staff  Sergeant  Friendly  is  preparing  a lesson  on  personnel  record  keeping  in  regard 
to  PERSCOM  FORM  362-A  (REV  1/9/75).  The  individual  serviceman’s  record  of  training  is 
kept  on  this  form  which  includes  entries  pertaining  to  military  training  courses,  USAFI 
courses,  and  courses  completed  in  civilian  schools  as  part  of  the  serviceman’s  career 
enhancement  program. 


Sergeant  Friendly  wants  to  use  Computer  Assisted  Instruction  so  that  he  can 
present  a variety  of  examples  that  illustrate  the  basic  categories  of  entries  to  be  made  on 
this  form.  In  CAI  he  can  occasionally  check  to  see  if  the  student  has  grasped  the  basic 
category.  If  the  student  has,  the  CAI  can  advance  him  to  another  c.a-egory;  if  not,the 
student  can  be  given  remediation  until  he  understands  the  ca.egory. 


At  the  same  time.  Sergeant  Friendly  wants  to  give  the  student  practice  in  actually 
making  the  entries  on  the  form.  The  sergeant  cannot  decide  how  best  to  do  this. 


LCDR  Moore,  the  navigator  aboard  the  Polaris  submarine  USS  GEORGE  WASHINGTON, 
has  just  been  relieved  as  Officer  of  ti.e  Deck  on  the  second  dog  watch.  After  stopping  by 
the  galley  for  a fresh  cup  of  coffee,  LCDR  Moore  goes  to  the  Communications  Room.  A 


I 
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special  stand-alone  CAI  terminal  is  kept  in  the  Comm  spaces  for  top  secret  work,  since  the 
Comm  spaces  are  among  the  most  higly  secured  areas  of  the  submarine.  The  terminal  is 
one  of  the  new  stand-alone  types  developed  for  use  aboard  Polaris  submarines  which 
must  maintain  tight  two-way  communication  silence  during  patrols. 

LCDR  Moore  checks  out  a CAI  pack  from  the  duty  Classified  Materials  Control  Officer 
and  takes  the  pack  to  the  terminal.  Moore  plugs  the  pack  in  and  keys  in  his  personal 
student  combination  on  the  terminal  to  log  in  and  activate  the  courseware. 

This  is  the  twelfth  in  a series  of  thirty  lessons  on  combat  command.  In  this  lesson 
LCDR  Moore  has  command  of  the  submarine  during  a simulated  hunter-killer  operation.  He 
will  encounter  an  enemy  submarine  of  the  same  type  and  engage  the  enemy  in  combat.  In 
his  first  such  simulated  engagement,  LCDR  Moore  lost  his  submarine  in  the  first  five 
minutes  of  battle.  But  he  has  improved  through  the  tutoring  of  the  CAI  system  and  got  a 
rating  of  750  out  of  1000  on  his  last  engagement. 
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Appendix  II.C.l 

SPECIAL  LETTER  TO  MILITARY  PARTICIPANTS 


[Name  of  participant] 
[Address  of  participant] 

Dear  [Title  and  last  name]: 


Grants  Office  of  Contracts  anc 

-.nrts  fro:  r 

that  you  will  respond  promptly  In  any  case.  ' 


Sincerely, 

/s/Monty  C.  Stanford 
for/  Bill  Mann,  ISI 

Rick  Carlson,  ASC 


Tom  Martin,  ASC 
Monty  Stanford,  ASC 
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Appendix  III.A 

ROUND  ONE  QUESTIONNAIRE 


You,  the  experts,  have  general  attitudes  that  will  be  contributing  to  your  specific 
responses  throughout  the  study.  We  need  to  find  out  what  those  attitudes  are  so  we  can 
find  a common  ground  if  one  exists,  or  at  least  can  better  understand  your  responses. 
The  following  trade-off  questions  attempt  to  tap  those  attitudes.  Place  ^ |V  next  to  the 
alternative  that  best  characterizes  your  opinion  and  check  marks  next  tp^  the^  other 
alternatives  you  would  be  willing  to  accept  for  purposes  of  consensus.  If  there  are  other 
trade-off  questions  we  should  have  asked,  feel  free  to  suggest  them.  Write  us 
explanations  if  you  think  it  might  help.  Remember  that  the  context  is  military  CAI/CMI. 


m 
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1.  The  user  interface  should  be 

a)  easy  to  use  (even  if  this  means  limiting  system 

capability) 

b)  powerful  (even  if  th's  makes  it  hard  to  use) 

2.  The  terminal  should  be 

a)  usable  for  a wide  range  of  tasks  (text-editing, 

programming,  etc.) 

b)  intended  specifically  for  CAI/CMI 

3.  The  system  should 

a)  adjust  to  the  user  (even  if  this  is  expensive 

computerwise) 

b)  have  the  user  adjust  to  it  (even  if  this  is  expensive 

humanwise) 

4.  The  user  interface  should  be  primarily 

a)  built  into  the  hardware 

b)  located  in  the  software 

5.  The  system  should  be 

a)  innovative  (even  if  it  sometimes  doesn’t  work) 

b)  reliable  (even  at  the  expense  of  discouraging 

innovation) 

6.  The  terminal/computer  resource(s)  should  be 
a)  _______  stand-alone 
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7.  The  design  philosophy  should  be  oriented  toward 

a)  maximizing  benefits  (even  where  costs  are  high) 

b)  minimizing  costs  (even  where  benefits  are  slight) 


45 
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The  next  fifteen  questions  deal  with  ranges  of  users,  tasks,  and  learning 


environments.  Once  again  place  an  "x"  next  to  the  alternative  that  best  characterizes 


your  opinion  and  a check  mark  next  to  alternatives  you  will  accept  fo£ 


consensus.  We  have  asked  you  to  try  to  explain  what  you  think  each  question  is  getting 


at  and  how  you  would  like  subsequent  rounds  to  probe  in  greater  depth.  Feel  free  to 


write  on  the  backs  of  pages  or  to  enclose  additional  pages. 


8.  The  terminal  should  be  locatable 


in  the  user’s  preferred  environment 


(even  if  help  is  not  readily  available) 


only  where  help  is  readily  available 


(even  if  this  is  not  the  user’s  preferred 


environment) 


9.  Military  personnel  vary  greatly  in  their  intellectual 


abilities.  Five  to  ten  years  from  now  (1980-1985),  the  greatest-, 


payoff  will  come  from  user  interfaces  intended  for 


the  total  range  of  user  intellectual  competencies 


the  more  intelligent  users 


users  of  average  intelligence 


10.  Military  personnel  vary  greatly  in  their  motivation  to  learn. 


Five  to  ten  years  from  now  (1980-1985),  the  greatest  payoff 


will  come  from  user  interfaces  intended  for 


the  total  range  of  user  motivations 


the  more  motivated  users 
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c)  users  of  average  motivation 

11.  User  interfaces  are  needed  for  many  aspects  of  CAI/CMI  activity. 
Five  to  ten  years  from  now  (1980-1985),  the  greatest  payoff 
will  come  from  user  interfaces  intended  for 

a)  the  total  range  of  system  activities 

b)  primarily  courseware  development 

c)  primarily  research  into  CAI/CMI  usage 

d)  primarily  student  learning 

12.  Computerized  systems  vary  greatly  in  the  portion  of  the 
teaching  load  that  they  carry.  Five  to  ten  years  from  now 
(1980-1985),  the  greatest  payoff  will  come  from  user  interfaces 
intended  for 

a)  the  total  range  of  CAI/CMI 

b)  primarily  CAI  (Where  they  carry  most  of  the  load) 

c)  primarily  CMI  (Where  they  carry  only  part  of 

the  load) 

13.  Military  training  courses  vary  greatly  in  the  time  it  takes  for 
students  to  complete  them.  Five  to  ten  years  from  now  (1980- 
1985),  the  greatest  payoff  will  come  from  user  interfaces 
intended  for 

a)  the  total  range  of  course  durations 

b)  those  requiring  a week  or  more 

c)  those  requiring  less  than  a waf»k 


Policy  Study  on  CAI  Terminals  47 

Appendix  III.A  - Round  One  Questionnaire 


14.  Military  training  tasks  vary  greatly  in  the  degree  to  which 
they  have  security  clearance  requirements.  Five  to  ten 
years  from  now  (1980-1985),  the  greatest  payoff  will  come 
from  user  Interfaces  intended  for 

a)  the  total  range  of  security  clearances 

b)  where  the  material  is  unclassified 

c)  where  the  material  is  classified  below  secret 

15.  Military  training  tasks  vary  greatly  in  the  need  for  spontaneous, 
free-form  student  responses.  Five  to  ten  years  from  now 
(1980-1985),  the  greatest  payoff  will  come  from  user 
interfaces  intended  for 

a)  the  total  range  of  user  responses 

b)  those  where  spontaneous,  free-form  responses 

are  important 

c)  those  where  spontaneous,  free-form 

responses  are  not  important 

16.  Military  training  tasks  vary  greatly  in  the  need  for 
non-canned,  student-tailored  courseware.  Five  to  ten  years 
from  now  (1980-1985),  the  greatest  payoff  will  come  from 
user  interfaces  intended  for 

a)  the  total  range  of  student -tailored  coursewar  e 

b)  those  where  non-canned,  student-tailored  course- 

ware is  important 
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c)  those  where  non-canned,  student-tailored  coiirse- 

ware  is  not  important 

17.  Military  training  tasks  vary  greatly  in  the  need  for  realistic 
visuals  (color,  three-dimensions,  shading,  movement).  Five 
to  ten  years  from  now  (1980-1985),  the  greatest  payoff  will 
come  from  user  interfaces  intended  for 

a)  the  total  range  of  visuals 

b)  where  realistic  visuals  are  important 

c)  where  realistic  visuals  are  not  important 

18.  Military  training  environments  vary  greatly  in  the  access- 
ability  of  an  instructor.  Five  to  ten  years  from  now 
(1980-1985),  the  greatest  payoff  will  come  from  user 
interfaces  intended  for 
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c) 


where  the  power  supply  is  available  outside  the 


terminal 

20.  Military  training  environ. rents  vary  greatly  in  the  access- 
ability  of  communication  bandwidth  (via  lines  or 
broadcast  frequencies).  Five  to  ten  years  from  now 
(1980-1985),  the  greatest  payoff  will  come  from  user 
interfaces  intended  for 

a)  th‘,'  total  range  of  bandwidth  accessability 

b)  where  no  outside  communication  is  possible 

c)  wtiere  narrow-band  (voice-grade  telephone) 


d) 


communication  is  possible 
where  broad-band  (cable-television) 


communication  is  possible 

21.  Military  training  environments  vary  greatly  in  the  need 
for  lightweight  equipment.  Five  to  ten  years  from  now 
(1980-1985),  the  greatest  payoff  will  come  from  user 
interfaces  intended  for 

a)  the  total  range  of  terminal  weights 

b)  where  the  terminal  weighs  less  than  thirty 


c) 


pounds 

where  the  terminal  weighs  thirty  pounds 


or  more 

22.  Military  training  environments  vary  greatly  in  the  duration 


now  (1980-1985),  the  greatest  payoff  will  come  from 
user  interfaces  intended  for 

a)  the  total  range  of  learning  session  durations 

b)  where  the  learning  session  lasts  less  than  an  hour 

c)  where  the  learning  session  lasts  an  hour 


or  more 


23.  Military  training  environments  vary  greatly  in  the  grouping  of 
stuaents.  Five  to  ten  years  from  now  (1980-1985),  the 
greatest  payoff  will  come  from  user  interfaces  intended  for 

a)  the  total  range  of  student  groupings 

b)  where  students  are  in  close  proximity 

c)  where  students  are  not  in  close  proximity 


s i 


So  far  sixteen  of  the  twenty-five  questionnaires  have  been  returned.  Eight  of  the 
twenty-three  questions  reflect  consensus,  i.e.,  a single  alternative  receiving  a 
preponderance  of  the  x’s  and  checks.  These  are  represented  below  with  the  favored 
alternative  first,  the  second-most  favored  alternative  last,  and  the  response  tally  in  the 
middle.  A vote  of  (10+i  vs  0+2)  means  that  the  favored  alternative  received  ten  x’s  and 
one  check  while  the  second-most  favored  alternative  received  two  checks.  The  eign. 
points  of  consensus  are: 


l a;  easy  to  use  (10+1  vs  1+1)  powerful 
3a;  adjust  to  user  (11+2  vs  1-0)  user  adjust 
5b;  reliable  (9+4  vs  2+0)  innovative 
7a;  maximize  benefits  (8+1  vs  3+1)  minimize  costs 
8a;  in  user’s  preferred  environment  (11  + 1 vs  3+1) 
where  help  is  available 

lid;  primarily  student  learning  (12+0  vs  3+4) 


range  of  research,  coursewriting,  and  learning 


13a;  range  of  course  durations  (iO+0  vs  3+0) 


duration  less  than  a week 


18b;  where  instructors  are  not  readily  available 


(10+1  vs  3+7)  range  of  instructor  availability 
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For  eight  of  the  questions,  opinions  were  split  between  a choice  of  the  total  range 
of  something  and  a single  one  of  the  alternatives.  Theses  are  represenfed  below  with  the 
range  first,  the  single  alternative  last,  and  the  response  tally  in  the  middle. 

9;  user  intellectual  competency  range  (11+2  vs  4+4) 
average  intelligence 

10;  user  motivation  range  (7+2  vs  4+5)  average 
motivation 

14;  security  clearance  range  (7+2  vs  5+1)  unclassifitd 

15;  spontaneous,  free-form  response  range  (8+5  vs  6+1) 
is  unimportant 

16;  non-canned,  student-tailored  courseware  range 
(8+4  vs  8+2)  is  important 

17;  realistic  visuals  range  (7+4  vs  4+3)  is  important 

19;  location  of  power  supply  range  (5+2  vs  7+0) 
outside  terminal 

22;  length  of  learning  session  range  (7+2  vs  7+1)  less 
than  an  hour 


For  seven  of  the  questions,  no  pattern  of  responses  emerged.  The  tallies  and 
comments  for  separate  items  follow. 
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General  comments  and  contributor: 

I continue  to  be  totally  mystified  by  this  project.  The  kind  of  information  that 
seems  to  interest  you  is  worthless  for  olanning  purposes.  Maybe  your  second  round 
(terminal  I/O  features)  will  be  more  metiningful,  but  I rather  expect  questions  such  as 
"which  is  more  important,  local  editing  or  •graphics?"  Not  that  I don’t  think  that  reasonable 
questions  could  be  constructed—l  just  thii..k  you  are  trying  something  that  is  literally 
impossible,  in  the  same  sense  as  violating  the  second  law  of  thermodynamics."  (Sherwood) 

"I’d  prefer  that  we  not  use  the  word  terminal,  particularly  as  it  could  mean  a 
stand-alone  system—why  not  display?"  (Bork) 

"The  function  of  the  scenarios  is  not  clear"  (Wexler) 

The  following  entries  will  deal  individually  with  each  first  round  question  and  the 
responses: 


****Question  One:  The  user  interface  should  be  a)  easy  to  use(even  if  this 
means  limiting  system  capability)— 10"x",  1 check — b)  powerful  (even  if  this  makes 
it  hard  to  use)  l"x",  1 check — 

COMMENTS:  Neither  (a  or  b)-  the  interface  must  be  so  powerful  that  it  is  easy 
to  use.(Sherwood)  "If  not  easy  to  use,  then  at  least  each  activity  should  be 
self-evident  in  context.  The  terminal  must  not  interfer  with  learning." 

"These  alternatives  are  not  incompatible.  (Dean)  Current  software 
developments  permit  both  types  of  user  interface  to  be  supported  by  a single 
system.  False  dichotomy.  (Ford) 

"but  more  power  can  be  "uncovered"when  needed."  (Zinn) 


♦♦♦♦Question  two:  The  terminal  should  be 

a)  usable  for  a wide  range  of  tasks  (text-editing,  programming,  etc.)  8"x",  0 checks— 

b)  intended  specifically  for  CAI/CMI.  5"x",  2 checks— 

COMMENTS:  "The  cost  benefits  of  volume  production  are  well  known. 
Terminals  should  not  be  exclusive  military  devices."  (Dean) 


♦♦♦♦Question  three:  The  system  should 

a)  adjust  to  the  user(even  if  this  is  expensive  computer  wise)  H"x"  , 2 checks — b) 
have  the  user  adjust  to  it  (even  if  this  is  expensive  humanwise)  l"x",  0 checks — 
COMMENT:  All  systems  are  adjusted  to  users  and  users  must  adjust 


'mm 


K . • 
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themselves  to  all  systems.  The  question  should  be  one  of  determining 
tradeoffs  for  specific  situations.  (Ford) 


♦♦♦♦Question  four:  The  user  interface  should  be  primarily 

a)  built  into  the  hardware  5V,  1 check—  b)  located  in  the  software  5 x , 2 

check— 

rnkA/iFWTC*  "Rnfh  about  eouallv-fclse 


"Question  really  is  one  of  engineering  design.  I suspect  that 
technology  will  force  the  answer.  (Dean) 

"This  requires  a technical  background  I don’t  care  where  it  is  located  as 
long  as  it  works."  (Hall) 

"a-When  economics  are  sure  to  be  realized  b-Where  flexibility  should 
be  maintained"  (Zinn) 


b)  reliable  (even 


♦♦♦♦Question  five:  The  system  should  be 

a)  innovative  (even  if  it  sometimes  doesn’t  work)  2'V,  0 checks— 
at  the  expense  of  discouraging  innovation)  9"x",  4 checks 

COMMENTS:  "These  and  similar  questions  are  meaningless.  Any  real-lite 
implementation  must  strike  a balance,,  and  it  is  not  possible  even  to  give  an 
■in/-iinaUnn  in  nnp  dirpction  or  another."  (Sherwood) 


"Innovative  during  development  phases  , but  reliable  when  it  is  put  into 
operations  with  students."  (Hall) 

"Quite  important  (reliabi''V)  operational  setting,  usually  is  first 
question  asked  by  military,  justifiable  or  not."  (Nawrocki) 

"Both-  I don’t  see  these  as  either/or  questions.  This  is  like  asking  if 
you  like  apples  or  oranges."  (Bork) 


♦♦♦♦Question  six:  The  terminal/computer  resource(s)  should  be 

a)  stand  alone  7"x",  3 checks-  b)  time-shared  6"x",  3 checks 

COMMENTS:  a-Where  logistics  demand  it,  b-for  maximum  cost 

where  possible(Dean)  "for  CAI/CMI  activities-time  shared  among  CAI/CMI 

users,  but  not  with  administrative  functions.  (Hall) 

"former  easier  to  implement  given  current  military  structure  and 
preferences,  though  latter  may  be  preferable  to  reduce  inter/intra  service 
redundancy."  (Nawrocki) 


Policy  Study  on  CAI  Terminals 
Appendix  III.B  - Round  One  Summary 


55 


"Most  of  the  time,  (stand  a'one)  but  with  possible  access  to  a large 
computer  occasionally  for  computation  or  large  data  base."  (answered 
reluctantly  perhaps,  but  I am  thinking  of  10  years  from  now.)  (nawrocki) 

"A  blanket  recommendation  is  not  possible."  (Ford) 


****Question  seven;  The  design  philosophy  should  be  oriented  toward 

a)  maximizing  benefits(even  where  costs  are  high)  8"x",  1 check — b)  minimizing 
costs  (even  where  benefits  are  slight)  3"x"  , 1 check— 
any  real-life  implementation  must  strike 

"latter  better  until  benefits  can  be  determined  accurately." 

"Some  suitable  mix."  (Bork) 

"The  intent  will  always  be  to  maximize  certain  benefits  and  to  minimize 
certain  costs.  Again,  requires  situation  specific  tradeoffs.  (Ford) 


♦♦♦♦Question  eight:  The  terminal  should  be  locatable 

a)  in  the  user’s  preferred  environmert(even  if  help  is  not  readily  available)  ll"x",  1 
check—  b)  only  where  help  is  readily  available  (even  if  this  is  not  the  user’s 
preferred  environment)  3"x",  1 check 

COMMENTS:  "except  for  initial  aid,  the  device  and  the  programs  can  be  made 
to  work  well  with  no  "help"."  (Bork) 


♦♦♦♦Question  nine  Military  personnel  vary  greatly  in  their  intellectual  abilities.  Five  to  ten 
years  from  now,  the  greatest  payoff  will  come  from  user  interface  intended  for: 

a)the  total  range  of  user  intellectual  competencies— H"x",  2 checks—  b)fhe  more 
intelligent  users--0"x",  1 check  c)users  of  average  intelligence— 4"x",  4 checks 

COMMENTS;  "This  is  basically  a question  of  vocation  vs  decision  skills.  Both 
are  necessary  for  maximum  benefit,  but  early  emphasis  will  be  on  vocation." 
(Nawrocki)  "Computer  materials  can  and  should  be  adaptable  to  a wide 
audience."  (Bork) 

"I  take  payoff  to  mean  a favorable  comparison  of  total  ’costs’  between 
using  skilled  human  teachers(if  available)  vs.  placing  major  emphasis  on 
CAI/CMI  in  the  following  kind  of  situation:  it  is  necessary  to  train  a single 
person  to  a specific  level  of  familiarity  or  expertise  in  a subject  or  on  an 
instrument  and  this  is  to  be  done  within  a prescribed  amount  of  (real)  time, 
(not  in  the  flavor  of  your  scenario).  The  time  and  proficiency  constraints  are 
to  be  jointly  satisfied".  (Wexler) 
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"most  desirable  features  of  terminals  seem  to  serve  both  b and  c 

(Zinn) 


«**Question  ten;  Military  personnel  vary  greatly  in  their  motivation  to  learn.  Five  to  ten 
years  from  now,  the  greatest  payoff  will  come  from  interfaces  intended  for: 

a)the  total  range  of  user  motivations-7"x",  2 checks  b)the  more  motivated 
users-2"x",  0 checks  c)users  of  average  motivation— 4"x",  5 checks 

COMMENTS:  "Instruction  s/b  written  to  motivate  the  user. 

(Weike-Glasser-Miller) 

"Motivation  may  be  an  irrelevant  question  in  military  environments. 
(Nawrocki) 

"The  highly  motivated  individual  can  learn  In  a wide  variety  of 
ways — real  learning  problems  come  from  those  not  so  motivated.  (Bork) 

"The  less  well  motivated."  (Anastasio) 


♦♦♦♦Question  eleven:  User  interfaces  are  needed  for  many  aspects  of  CAI/CMI  activity. 
...The  greatest  payoff  will  come  from  user  interfaces  intended  for: 

a)the  total  range  of  system  activities-3"x",4checks  b)primarily  ^^courseware 
development-0"x",2checks  c)primarily  research  into  CAI/CMI  usages  l"x",  1 check 

d)primarily  student  learning--12"x",0checks  t ir  * 

COMMENTS:  "tricky  question  to  answer.  Interpreted  this  in  terms  of  ultimate 
goal,  though  student  learning  will  not  be  significant  without  others. 

(Nawrocki) 

"small  number  of  terminals  of  special  designfif  necessary)  could  serve  b 
and  c — (Zinn) 


♦♦♦♦Question  twelve:  Computerized  systems  vary  greatly  in  the  portion  of  the  teaching 
load  that  they  carry.  ...the  greatest  payoff  will  come  from  user  interfaces  intended  for: 

a)the  total  range  of  CAI/CMI-8"x",3  checks  b)primarily  CAKWhere  they  carry  most  of 
the  load)  4"x",  2 checks  c)primarily  CMKWhere  they  carry  only  part  of  the  load) 
4”x" 2 chscks 

COMMENTS:  "I  don’t  l-ke  this,  but  it  seems  to  be  the  case.  CAI  likely  to  be 
relatively  limited  to  material  requiring  simulation/gaming,  but  still  hard  to  say 
with  any  certainty."  (Nawrocki) 

"I  regard  this  as  a partially  artificial  distinction,  reflecting  our  current 
relatively  primitive  abilities."  (Bork) 
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****Question  Thirteen:  Military  training  courses  vary  greatly  in  the  time  it  takes  for 
students  to  complete  them.  ...the  greatest  payoff  will  come  from  user  interfaces  intended 
for: 

a)the  total  range  of  course  durations  lO'V',  0 checks  b)those  requiring  a week  or 
more-0"x",  1 check  c)those  requiring  less  than  a week--3"x",  0 checks 

COMMENTS:  "for  modules  of  courses  requiring  20  hours  or  less  of  study-our 
research  Indicates  that  20  hours  is  maximum  for  study  length."(Dean)  "What  is 
magical  about  a week?"(Bork) 

user  interfaces  should  respond  to  rather  than  determine  the  aspects  I 
have  not  marked,  e.g.  if  training  of  short  duration  is  better  tailored  to 
trainee  then  terminal  should  be  portable,  etc.  (no  responses  given)  (Zinn) 


♦♦♦♦Question  fourteen:  Military  training  tasks  vary  greatly  in  the  degree  to  which  they 
have  security  clearance  requirements...the  greatest  payoff  will  come  from  user  interfaces 
intended  for 

a)the  total  ranp  of  security  clearances  7"x",  2 check  b)where  the  material  is 
unclassified  5 x , 1 check  c)where  the  material  is  classified  below  secret — 0 "x",2 
checks 

COMMENTS:  b-Classified  material  would  be  a management  problem." 

(Weike-Glasser-Miller) 

"Mostly  because  much  equipment  and  info  referred  to  will  be  classified 
to  some  extent.'Yresponse  a"x")  (Nawrocki) 

Why  is  it  an  issue?  Is  the  worry  that  that  the  system  might  not  be 
secure?"(Bork) 

"If  classified  material  should  be  handled  in  automated  system,  then 
terminal  can  designed  accordingly  to  assure  security(no  response  indicated) 
(Zinn) 


♦♦♦♦Question  fifteen:  Military  training  tasks  vary  greatly  in  the  need  for  spontaneous 
free-form  student  responses.  ...the  greatest  payoff  will  come  from  user  interfaces 
intended  for:  a)the  total  range  of  user  responses-8"x"  5 checks  b)those  where 
spontaneous,  free-form  responses  are  important-2"x",  2 checks  c)those  where 
spontaneous,  free-form  responses  are  not  important— 5"x",  1 check 

COMMENTS:  "Education  should  adapt  to  student,  not  vice  versa."  (Bork) 

'Author  of  training  material  should  not  be  constrained,  but  only  advised 
of  costs  or  other  considerations  which  might  favor  less  spontaneous 
response."  (Zinn) 
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m thp  need  for  non-canned  , 
**«Question  sixteen:  Military  trairung  come  from  user  interfaces 

student  tailored  courseware,  -the  greai.si  p y 

intended  for:  . * ^ t tailored  courseware--8  "x",A  checks  b)those  where 

a)the  total  range  of  student  tailor  imDortant-8"x",2  check  c)those  where 





iw  thp  n66cl  for  roslistic 

♦♦♦♦Question  seventeen:  Military  the'greatest  payoff  will  come  from 

visulfcolor.  three-dimensions,  shading,  movement).  ...the  g 

user  interfaces  intended  for  _ visuals  are  important  A 

a)the  total  range  of  visuals-7  x , A ° 7 . . 2 "x",  Ochecks 

"x",  Schecks  c)where  realistic  visuals  are  >P  . ^ ^ personal  bias  is 

■ COMMENTS:  "Who  know  7""  *^^,^77rely  on  a ^ 

r S.n-I,v  e„ecv,ve. 


/ Kl ...  . I ^ 


••but  perhaps  experimental  work  here  will  show  that  not 
equally  valuable."  (Bork) 

•someti.es  tine  drawingstand  animated  drawings  are 
than  photos."  (Zinn) 


all  of  these  are 


more  important 


....Question  eighteen:  Military  traini^^  '^^^0^1,^  WshLIs  ilndedTon”'’'''''' 
an  instructor.  ...the  greatest  payo  wi  . g»  ..  j checks  b)where  instructors  are 

:>rre^i:;  m;tr.tors  are  readily  aeaiiahte  3 y , 

° ''’"'comments:  ■note--Pjwous  « " tea^ly'lPture  CAI 

:;:remf:ili  r'epuTe  t'ar  less  .rnagement."  (WeiKe-Giasser-Miller) 

■opinion:  probably  use  of  remote  terminals  will  come  into  its  own." 
(Nawrocki) 


"c— computers  may  play  lesser  role.  (Zinn) 
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****Question  nineteen:  Military  training  environments  vary  greatly  in  the  accessability  of  a 
power  supply,  ...the  greatest  payoff  will  come  from  user  interfaces  intended  for: 

a)the  total  range  of  power  accessability  5"x",  2 check  b)where  the  power  supply 
must  be  contained  in  the  terminal—TV',  1 check  c)where  the  power  supply  is 
available  outside  the  terminal-7"x",  0 checks 

COMMENTS:  "doubt  this  will  be  feasible  in  time  period  required-if  so  very  high 
utility"(response  c="x"  ' =check)--more  than  likely  will  emphasize  option  c 
with  gradual  use  of  option  b.fas  supolement  not  alternative)."  (Nawrocki) 

"in  a true  scale  power  will  still  make  considerable  demands."  (Bork) 

"training  decisionfno  response  indicated)  (Zinn) 


♦♦♦♦Question  twenty:  Military  training  environments  vary  greatly  in  the  accessability  of 
communication  bandwidths  (via  lines  or  broadcast  frequencies).  ...the  greatest  payoff  will 
come  from  user  interfaces  intended  for: 

a)the  total  range  of  bandwidth  accessability  6"x",  3 checks  b)where  no  outside 
communication  is  possible  l"x",  2 checks  c)where  narrow-band(voice-grade 
telephone)  communication  is  possible  2"x",4  checks  d)where  broad-band 
(cable-television)  communication  is  possible-3"x"  3 checks 

COMMENTS:  "this  problem  does  not  arise  with  stand  alone  systems,  one  of  the 
reasons  that  such  systems  will  be  important  in  the  future."  (Bork) 

"c-d  "if  economical,  then  interaction  with  instructors  via  cable  may  be 
preferable  to  computers."  (Zinn) 


♦♦♦♦Question  twenty-one:  Military  training  environments  vary  greatly  in  the  need  for 
lightweight  equipment.  ...the  greatest  payoff  will  come  from  user  interfaces  intended  for: 
a)the  total  range  of  terminal  weights  6"x",  2 checks  b)where  the  terminal  weights 
less  than  than  thirty  pounds— 5"x",  4 checks  c)where  the  terminal  weighs  thirty 
pounds  or  more— l"x",  2 checks 

COMMENTS:  "hard  to  imagine  a terminal  of  any  capacity  weighing  more  than 
30  lbs,  ten  years  from  now."  (Dean) 

"obviously  a technological  question.  Weight  important,  but  terminal 
effectiveness  more  important."  (Nawrocki) 

"not  the  most  critical  issue."  (Bork) 

"Portability  is  not  yet  an  issue"  (Anastasio) 


Policy  Study  on  CAI  Terminals 
Appendix  III.B  - Round  One  Summary 


60 


♦♦♦♦Question  twenty-two:  Military  training  environments  vary  greatly  in  the  duration  of  an 
average  learning  session.  ...the  greatest  payoff  will  come  from  user  interfaces  intended 
for: 

a)the  total  range  of  learning  session  duration  7"x",  2 checks  b)where  the  learning 
session  lasts  less  than  an  hour-7"x",  1 checks  c)where  the  learning  session  lasts  an 
hour  or  more-l"x",0  checks 

COMMENTS:  "much  more  than  an  hour  will  bore  or  at  least  tire  the  student." 
(Weike-Glasser-Miller) 

"1-2  hours  max."  (Dean)  "clearly  varies  with  content  of  course,  need, 
etc.  Not  a particularly  useful  question."(Nawrocki) 

"Training  decision  " (Zinn) 


♦♦♦♦Question  twenty-three:  Military  training  environments  vary  greatly  in  the  grouping  of 
students,  ...the  greatest  payoff  will  come  from  user  interfaces  intended  for: 

a)the  total  range  of  student  groupings  6"x",  1 check  b)where  students  are  in  close 
proximity  4"x",  3 checks  c)where  students  are  not  in  close  proximity  — 3"x",  2 
checks 

COMMENTS:  "students  should  interact  with  terminals  alone!  should  discuss 
course  material  with  other  students  offline!"(Dean) 

"strictly  opinion,  current  thinking  and  planning  tends  to  emphasize 
option  b."(Nawrocki) 

"small  groups  of  2-3  offer  great  advantages  advantages  for  many  types 
of  work-student  learn  much  from  interacting  with  each  other."(Bork) 
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Appendix  III.C 

ROUND  TWO  QUESTIONNAIRE 

Hi,  enclosed  are  the  results  of  round  one  of  the  1980-  1985  military  CAI/CMI  study 
and  the  questions  for  the  second  round.  The  response  rate  was  over  sixty  percent  (16 
out  of  25)  and  it  took  a long  time  for  responses  to  trickle  in.  This  time  we  plan  to  reward 
you  (25  dollars)  for  responding  promptly.  (If  you  have  not  yet  returned  round  one,  you 
may  want  to  do  so  now.) 

Last  time  we  told  you  that  round  two  would  get  down  to  features.  Instead  we 
decided  to  use  this  round  as  a bridge  between  the  first  round  and  the  features  round. 
The  questions  fall  into  three  groups—  Whether  or  not  an  investment  in  terminal 
development  is  likely  to  le^d  to  the  greatest  payoff,  preparations  for  round  three  feature 
identification  questions  and  further  probing  of  round  one  issues.  We  will  get  results  to 
you  as  soon  as  responses  have  been  collected  and  analyzed,  and  we  strongly  encourage 
you  to  be  verbose  in  your  responses.  Remember  that  this  is  more  a discussion  to  help 
you  influence  us  than  a reliable  sampling  of  some  homogeneous  population. 


1.  Do  you  think  that  an  investment  in  the  development  of  new  terminals  for 
CAI/CMI  will  have  a significant  payoff?  What  are  some  of  the  reasons  underlying  your 
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2.  Do  you  think  that  five  to  ten  years  from  now  the  mainstream  terminal  vendors 
will  be  responsive  to  the  needs  of  the  military  CAI/CMI  users  (assuming  no  concerted  push 
by  the  military)?  What  terminal-related  needs  are  likely  to  go  unmet  without  a concerted 
push?  What  features  do  you  think  will  be  available  by  then? 


3.  It  can  be  argued  that  factors  unrelated  to  terminals  are  responsible  for  holding 
back  the  advancement  of  CAI/CMI.  In  what  areas  (other  than  terminal  development'  do 
you  think  that  an  R&D  investment  would  more  significantly  advance  the  1980-19ii5  state 
Of  the  art  in  CAI/CMI?  Why? 


It  is  likely  that  the  design  specifications  for  CAI/CMI  terminals  resulting  from  this 
project  will  fall  into  a hierarchy.  At  the  top  level  are  components  that  can  be 
incorporated  at  the  time  of  procurement.  At  the  bottom  level  are  components  that  can  be 
incorporated  (plugged-in)  by  the  end  user.  Components  can  be  input  devices,  displays, 
storage  media,  or  processors.  In  the  next  round  we  intend  to  find  out  your  priorities 
regarding  which  components  should  be  at  which  levels.  Right  now  we  need  to  find  out  a 
little  abou*  the  top  through  bottom  levels. 


ytics^ 
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4.  What  components  do  you  feel  could  be  standard  across  all  terminals?  How 
strongly  do  you  feel  about  each? 


5,  What  components  do  you  feel  should  be  optional  in  which  situations  so  that  they 
can  be  added  on  by  the  end  user.  How  strongly  do  you  feel  about  each? 


6.  In  anticipation  of  the  next  round,  what  specific  components  do  you  want  us  to 
ask  about?  Is  there  any  other  advice  you  want  to  g ve  us? 


As  indicated  in  the  summary  of  the  first  round,  eight  questions  had  responses 
balanced  between  a total  range  of  something  and  a particular  option.  Two  of  the 
questions  related  to  users  of  average  intelligence  and  average  motivation.  We  would  like 


( 
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to  ask  those  questions  again,  reworded  slightly,  to  be  sure  we  are  reading  you  ccrrectly 
Elaborate  upon  your  answers  if  it  will  help. 


?1 


7.  While  it  is  true  that  the  total  range  of  user  intellectual  capabilities  must  be 
considered  when  designing  CAl/CMl  user  interfaces,  do  you  agree  that  priority  should  be 
given  to  interfaces  intended  for  users  of  average  or  below  average  iiiielligence?  Why? 


8.  While  il  is  true  that  the  total  range  of  user  motivational  levels  must  be 
considered  when  designing  CAl/CMl  user  interfaces,  do  you  agree  that  priority  should  be 
given  to  interfaces  intended  for  users  of  average  or  below  average  motivation?  Why? 


9.  What  bottlenecks  make  it  difficult  for  users  of  average  or  below  average 


intelligence  to  use  CAl/CMl  today? 


Policy  Study  on  CA!  Terminals 
Appendix  III.C  - Round  Two  Questionnaire 


65 


10.  What  bottlenecks  make  it  difficult  for  users  of  average  or  below  average 
motivation  to  use  CAI/CMI  today? 


1 1.  In  operational  terms,  what  does  an  "easy  to  use"  terminal  look  like?  (e.g.,  limited 
choice  in  options,  lack  of  an  alphanumeric  keyboard,  hardware  to  lock  out  features  that 
aren't  applicable). 


As  indicated  in  the  summary  of  the  first  round,  a number  of  the  questions  elicited 
consensus.  The  user  interface  should  be  reliable,  be  easy  to  use,  adjust  to  the  user,  be  in 
his  preferred  environment,  and  not  be  dependent  upon  readily  available  instructors.  We 
need  to  draw  out  the  implications  that  flow  from  these  priorities  and  be  certain  we  are 
reading  you  correctly.  The  questions  will  tackle  acquisition  of  user  interface  skill  i»nd 
support  backing  up  use  of  the  interface.  Elaborate  upon  your  answers  if  it  will  help. 


l' 
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12.  How  much,  when,  and  primarily  from  whom  should  the  beginning  user  learn 
about  the  interface?  (e.g.,live  instructors,  other  users,  remote  consultants,  software, 
written  guides) 


.r-* 
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13.  How  much,  when  and  primarily  from  whom  should  the  experienced  user  learn 
about  additional  aspects  of  the  interface? 


! 


14.  What  sorts  of  things  should  the  system  be  able  to  take  care  of  so  that  the  user 
does  not  have  to  learn  about  them  (e.g.,bad  telephone  connection,  spelling  errors,  error 
recovery...) 


15.  When  things  are  not  functioning  properly  and  the  system  cannot  assist  the 
user,  who  and  what  are  the  primary  sources  of  support  that  the  user  can  fall  back  on? 


I 
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16.  There  was  consensus  that  the  greatest  payoff  will  come  from  an  emphasis  upon 
student  learning  (rather  than  the  total  range  of  research,  course  writing,  and  learning). 
Are  we  at  a stage  where  enough  is  known  about  course  writing  and  learning  so  that 
reliable  and  effective  courseware  can  be  developed? 


17.  During  the  first  round,  a lot  of  questions  about  this  versus  that  priority  were 
asked.  A number  of  you  felt  this  was  unfair,  unnecessary,  or  could  not  be  done.  In  this 
round  we  have  continued  extracting  your  priorities,  preferences,  and  opinions  about  areas 
where  payoffs  are  most  likely.  By  now  you  probably  have  a personal  set  of  the  most 
central  priorities'  when  thinking  about  the  1980-85  military  CAI/CMI  user  interface.  What 


are  they? 
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APPENDIX  liLD 
ROUND  TWO  SUMMARY 

1.  Do  you  think  an  investment  in  the  development  of  new  terminals  for  CAI/CMI  will  have 
a significant  payoff?  What  are  some  of  the  reasons  underlying  your  opinion? 


COMMENT:  Most  respondents  feel  that  an  investment  in  terminal  development  will 
have  a significant  payoff  (Bork,  Ditzik,  Giunti,  Hall,  Kimberlin,  Rockway,  Sherwood, 
Zinn).  Reasoning  varied,  but  the  following  comments  are  generally  representative: 


Yes,  but  not  just  terminals  — also  stand-alone  systems  with  intelligence. 
Existing  te-'minals  have  been  designed  primarily  for  the  business  morket,  so 
are  not  ideal  for  education.  (Bork) 


Yes,  if  done  right.  First,  in  order  to  have  a significant  payoff  the  terminal 
must  be  general  enough  for  university,  public  school,  industrial,  and  military 
instruction.  However,  the  entire  CAI/CMI  system  must  be  applied  in  such  a 
way  not  to  discourage  or  threaten  the  instruction  by  its  use.  That  is  the 
system  must  be  adjunct  to  the  normal  instruction  not  a mainline  instructional 
system.  Plus,  the  terminal  itself  must  be  designed  from  ground  up 
incorporating  established  human  factor  requirements.  (Ditzik) 


Yes,  I do  think  an  investment  in  the  development  of  new  terminals  for  CAI/CMI 
could  have  significant  payoff.  The  major  reason  for  this  assumption  is  that 
there  are  many,  particularly  CAI  type,  applications  for  terminals  with  varying 
degrees  of  complexity.  These  could  be  designed  to  match  the  requirements 
of  the  particular  learning  tasks  involved  and  the  instructional  strategies  being 
implemented.  For  example,  the  current  PLATO  IV  plasma  panel  is 
over-designed  (and  too  expensive)  for  many  of  the  applications  for  which  it  is 
being  used.  Much  of  the  material  being  delivered  is  in  a programmed 

instructional  format  with  a multiple-choice  type  response.  It  would  be  much 
more  economical  to  use  a simple  responder  augmented  by  off-line  adjunct 
materials.  I accept  the  need  for  a family  of  terminals  with  each  clas;  being 
designed  to  meet  a particular  set  of  instructional  requirements.  Just  how 
many  classes  should  be  considered  and  what  their  functional  characteristics 
would  be  is  a matter  for  a more  detailed  analysis.  (Rockway) 


follows: 


No,  1 believe  that  commercial  vendors  will  develop  competitive  termirials  in 
response  to  market  forces.  With  possible  exception  of  Plato,  which  is  still 
unavailable,  as  far  as  image  and  audio  are  concerned  no  significant  terminal 
has  been  devi->loped  except  by  industry.  (Dean) 


I think  existing  terminals  can  do  far  more  than  they  are  being  required  to  do. 
In  my  view  courseware  design  is  far  more  crucial  than  design  of  a particular 
delivery  system.  (Merrill) 


COMMENT:  And  two  respondents  were  "lukewarm"  about  terminal  development: 


jtrikes  me  that  the  available  terminals  (current)  are  more  than  sufficient  for 
CMI  already  and  that  the  only  additional  technology  with  potential  utility  might 
be  audio  input/output  communication.  For  CAI,  major  "need  would  seem  to 
be  low  cost  sophisticated  graphics  terminal  (3-D,  color,  etc.)..  Also 
recognition  of  voice  or  written  input  might  be  worthwhile.  The  preceding, 
plus  increacod  portability  would  seem  to  have  highest  payoff  potential  based 
on  user  "complaints"  and  to  a lesser  extent,  current  data  on  learning 
processes  and  man-machine  communication.  (Nawrocki) 


The  PLATO  group  did  tinker  and  did  come  up  with  an  interesting  terminal 
configuration  (although  they  have  yet  to  meet  their  original  cost  projections). 
You  ask  if  similar  developments  are  likely  to  occur  in  the  near  future 
I (1980-5).  In  thinking  about  the  phrase  "significant  payoff"  I find  my  response 

to  be  a lukewarm  "maybe."  (Wexler) 

I 


2.  Do  you  think  that  five  to  ten  years  from  now  the  mainstream  terminal  vendors  will  be 
responsive  to  the  needs  of  the  military  CAl/CMI  users  (assuming  no  concerted  push  by  the 
military)?  What  terminal-related  needs  are  likely  to  go  unmet  without  a concerted  push? 
What  features  do  you  think  will  be  available  by  then? 
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Among  the  reasons  mentioned  were; 


COMMENT:  Most  respondents  feel  vendors  will  not  be  responsive  (7  NO,  1 YES,  3 
N0(?),  3 YES(?).  Military  and  civilian  CAI/'"MI  needs  are  essentially  similar  (Bork, 
Ditzik,  Ford).  Neither  market  is  large  enough  to  influence  vendors,  who  respond 
more  to  business  applications  of  terminals  (Bork,  Hall,  Zinn). 


Equipment  manufacturers  will  not  be  responsive  to  the  needs  of  military 
CAI/CMI  users  unless  a concerted  push  is  made.  Their  approach  in  the  past 
has  been  to  give  educators  the  very  best  station  that  the  bankers  and  airlines 
need  for  their  applications  and  let  the  educators  adjust  their  instructional 
materials  to  fit  the  existing  hardware.  The  military  has  very  specialized 
training  requirements  which  must  be  met  by  a specially  designed  terminal,  e.g., 
the  display  of  alectronic  diagrams,  symbols,  mathematic  equations,  special 
symbols  for  physics,  and  chemistry.  Terminals  must  be  capable  of  displaying 
any  material  which  can  be  printed  on  a page,  display  photographic  materials 
under  systems  control,  and  play  audio  material  under  systems  control.  These 
needs  are  likely  to  go  unmet  because  banks  and  rirlines  don’t  need  them. 
Unless  a concerted  push  is  made  it  is  likely  that  only  *he  display  ot  upper  and 
lower  case  characters  will  be  available  with  perhap;  some  very  rudimentary 
microfiche  retrieval  procedures  because  bankers  need  that  to  check  current 
balances  before  accepting  personal  checks.  (Hall) 


COMMENT:  Furthermore,  military  CAI/CMI  terminal  needs  are  not  well  defined  or 
expressed  (Nawrocki,  Ward).  However,  some  respondents  feel  that  a concerted 
push  from  the  military  could  influence  vendors  (Bork,  Ditzik,  Rockway). 


Vendors  will  be  responsive  if  the<'e  is  an  economic  payoff.  The  military 
should  continue  to  sponsor  R and  D to  lower  costs  and  must  make  volume 
purchases.  Obviously,  agreements  and  standardizations  regarding  military 
terminals  could  result  in  larger  volume  buys.  (Rockway) 


3.  It  can  be  argued  that  factors  unrelated  to  terminals  are  responsible  for  holding  back 
the  advancement  of  CAI/CMI.  In  what  areas  (other  than  terminal  development)  do  you 
think  that  an  R and  D investment  would  more  significantly  advance  the  1980-1985  state  of 
the  art  in  CAI/CMI?  Why? 
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COMMENT:  Almost  all  respondents  stressed  need  for  performance  demonstration  of 
CAI/CMI  and  need  for  emphasis  on  the  total  learning  environment.  Many 
respondents  noted  that  work  on  utilizing  potential  of  CAI/CMI,  especially  interactive 
programs,  and  research  on  learning/instruction  techniques  are  more  important  than 
R and  D of  terminals.  Typical  responsec  follow: 


This  was  addressed  in  the  Educom  conference  of  several  years  ago,  and 
discussed  in  the  report  of  that  conference.  The  single  major  factor,  in  my 
opinion,  is  the  scarcity  of  good  highly  interactive  learning  programs,  rewritten 
many  times  on  the  basis  of  extensive  student  use.  Partially  this  is  just  a lack 
of  experience  — we  have  much  to  learn  about  how  to  write  very  effective 
student  — computer  dialogue,  and  this  learning  must  involve  not  only  research 
but  considerable  experience.  Most  of  the  current  authors  are  still  in  a 
"textbook"  mode,  not  exploiting  the  full  capability  of  the  media.  Much 
additional  work,  too,  is  needed  with  authoring  systems  which  ease  the  task  of 
preparation  or  materials.  (Bork) 


My  advice  is  "save  your  money"  rather  than  R and  D.  I would  like  to  see 
production  experiments  built  around  state  of  the  art  gear  conducted.  (Dean) 


One  of  the  biggest  hurdles  in  advancing  CAI/CMI  is  providing  authoring 
procedures  which  are  simplified  yet  powerful  enough  to  allow  very 
sophisticated  instructional  strategies  to  be  employed  by  content  specialists 
who  are  not  high-powered  programming  specialists.  A specialized  authoring 
facility  could  be  developed  to  generate  input  data  which  could  then  be 
compiled  into  the  operating  language  of  any  operational  CAI/CMI  system.  This 
would  enhance  the  transportability  of  curriculum  from  one  system  to  another. 
(Hall) 


Courseware  design,  strategy,  content  analysis.  In  my  view:Delivery  system 
(terminal)  modification  makes  the  least  difference  to  learning  while 
modifications  in  strategy  and  content  structure  can  make  major  differences. 
Far  too  much  attention  to  the  "cosmetics"  of  instruction.  Far  too  little  effort 
on  the  substance  of  instruction.  (Merrill) 


The  broader  application  of  CAI/CMI  in  the  immediate  future  is  largely 
dependent  on  successful  (that  it  cost-effective)  demonstrations.  The  longer 
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term  success  of  the  area  would  appear  to  depend  on  improvements  in 
instructional  technology  as  well  as  reductions  in  the  cost  of  CAl/CMl 
hardware.  On  the  instructional  technology  side,  one  of  the  major 
requirements  is  the  improvement  in  instructional  strategies  to  capitalize  on 
the  flexibility  of  computerized  delivery  systems.  Another  is  to  develop 
authoring  aids  to  reduce  the  time  and  cost  of  course  design  and  instructional 
materials  development.  However,  even  if  no  improvements  were  made  in  the 
current  state-of-the-art  of  instructional  technology,  simply  lowering  the  cost 
of  CAl/CMl  systems  to  make  them  competitive  with  conventional  techniques 
would  do  as  much  as  anything  to  expand  applications.  (Rockway) 


R and  D funding  might  better  be  directed  towards  the  development  of 
"intelligent"  teaching  systems  for  both  students  and  teachers. 


From  the  point  of  view  of  a student  working  in  a conventional  CAl/CMl 
environment  there  simply  is  not  the  richness  or  flexibility  that  arises  in  a live 
human  teaching  environment.  The  type  of  adaptation  exhibited  by  a teacher 
who  notices  the  pattern  of  a student’s  responses  and  utterances,  and  then 
proceeds  to  make  appropriate  alterations  in  a curriculum  sequence  is  only 
clumsily  replicated.  This  lack  of  response  options  obviously  arises  from  the 
difficulty  (impossibility)  of  foreseeing  or  pursuing  all  the  threads  emanating 
from  a standard  curriculum  strand.  Perhaps  one  way  to  improve  the  situation 
is  to  provide  the  course  author/teacher  with  an  "intelligent  teacher’s  aid." 


The  situation  might  be  imagined  where  a teacher  preparing  a course  can  turn 
to  a nearby  aid  and  remark  that  "this  materiai  involves  concepts  Cl,  C2....and 

has  features  FI,  F2 and  is  in  the  following  general  relationships  with  what 

has  and  will  be  covered  Rl,  R2,....  The  aid  might  also  ask  questions  when 
something  unusual  appeared  such  as:  did  you  really  mean  to  associate  those 
concepts?  Are  not  these  features  incompatible?  Is  not  this  combination  of 
relationships  curious?  etc.  Then  let  the  teacher  leave  and  let  the  teaching  aid 
assume  the  responsibility  of  interacting  with  a student  when  difficulties  arise 
in  accordance  with  the  directives  and  associations  supplied  by  the  teacher. 
Thus  the  aid  should  make  strategic  (and  "intelligent")  use  of  the  (possibly 
loose)  information  provided  by  the  teacher.  It  may  simplify  course 
preparation  and  expand  the  range  of  treatable  situations.  (The  above 
description  is  awkward  and  needs  refining  but  hopefully  indicates  the  trend  of 
my  thoughts.)  (Wexler) 


Some  other  possibilities,  but  probably  wouldn’t  do  any  more  than  very 
significant  developments  in  terminals,  since  the  interface  with  the  user  affects 
all  components:  Instructional  science,  particularly  to  make  better  use  of  the 
dynamic  nature  of  the  computer-based  training  system,  including  attention  to 
the  development  of  learning,  self-testing  and  other  skills  in  the  learner. 
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Prescriptions  for  effective  training  materials,  in  order  that  the  first  draft  can 
come  closer  to  the  final  product  (cost  savings,  primarily) 

Incorporation  of  training  into  performance  systems,  including  use  of 
I simulations  for  practice  (what  degree  of  abstraction,  fidelity,  tutoring,  etc,?) 

and  monitors  of  performance  in  actual  operating  systems,  ('inn) 


A.  What  components  do  you  feel  could  be  standard  across  tH  terminals?  How  strongly  do 
you  feel  aoout  each? 


COMMENT  : Numbers  of  responses  follow  each  feature. 

CRT  Display  — 2 

Graphic  Display  --  4 

Still  Images  --  5 

Hardcopy  Output  --  2 

Interface  to  Equipment  — 2 

Keyboard  --  7 

Pointing  Input  --  5 

Stored  Audio  Output  --  4 


The  standard  basic  terminal  should  provide  the  following  features:  Display  of 
any  material  capable  of  being  presented  on  a standard  textbook  page  which 
includes  use  defined  graphics  and  Especial  characters,  keyboard  input,  light  pen 
or  touch  sensitive  input,  random  access  photographic  image  retrieval  and 
display,  and  random  access  audio  retrieval  and  display.  ! would  not  tolerate 
any  deviation  on  having  these  facilities  available  at  each  station.  Making  them 
a requirement  on  all  stations  has  two  advantages:  (1)  it  encourages  authors  to 
develop  more  sophisticated  and  richer  instructional  materials  because  the 
facilities  are  there.  (2)  Because  they  are  available  at  each  station  they  are 
less  expensive  than  if  they  were  produced  in  smaller  quantities.  (Hall) 


I don’t  favor  plug-in  media  — things  tend  to  be  written  primarily  for  minimal 
system,  so  most  programs  would  not  use  such  plug-in  facilities.  Simple 
graphic  input  should  be  standard  (like  that  in  TEK  4010s)  (Bork) 
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5.  What  components  do  you  feel  should  be  optional  in  which  situations  so  that  they  can 
be  added  on  by  the  end  user.  How  strongly  do  you  feel  about  each? 


COMMENT;  Numbers  of  responses  follow  each  feature. 


Still  Images  --  2 
i-Iardcopy  Output  — 3 
Stored  Audio  Output  — 2 
Interface  to  Equipment  — 6 
Videotape  — 3 
Large  Area  Screens  — 3 
Pointing  Input  — ' 5 
Audio  Input  " 5 
Special  Keyboards  — 2 
Computing  Power  — 2 


6.  In  anticipation  of  the  next  round,  what  specific  components  do  you  want  us  to  ask 
about?  Is  there  any  other  advice  you  want  to  give  us? 


COMMENT:  Most  comments  were  similar  to  Hall’s. 


I would  hope  that  the  next  round  of  questions  would  include  each  of  the 
components  that  I mentioned  as  being  requirements  for  each  station  to  find 
out  to  what  extent  my  opinions  are  held  generally  throughout  the  survey 
group.  (Hall) 


Those  I’ve  already  mentioned.  Avoid  "terminal"  see  If  the  word  "military" 
makes  any  difference.  (Bork) 
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Ask  for  specific  types  of  terminal  usage  anticipated,  extent  of  usage  and 
groups  using  terminals  asking  about  terminal  components  is  wrong.  You  want 
to  determine  usage  requirements  for  training,  counseling,  management,  etc., 
and  then  dovalop  a system  which  will  meet  those  requirements.  (Ford) 


Ask  about  all.  Perhaps  you  should  ask  for  cost  estimates  in  1980-5  for 
individual  components  --  which  might  reflect  their  plausibility  along  with 
estimates  on  what  will  constitute  a reasonable  terminal  cost  at  that  time  (in 
1975  dollars?)  (Wexler) 


7.  While  it  is  true  that  the  total  range  of  user  intellectual  capabilities  must  be  considered 
when  designing  CAI/CMI  user  interfaces,  do  you  agree  that  priority  should  be  given  to 
interfaces  intended  for  users  of  average  or  below  average  intelligence?  Why? 


COMMENT:  Consensus  was  achieved  (10  Yes,  3 No,  1 Other).  Most  comments  were 
similar  to  Wexler’s: 


Yes,  they  will  probably  constitute  the  largest  satisfiable  group.  I don’t  expect 
a great  increase  in  the  level  of  sophistication  of  interactive  dialogues  and  I 
expect  brighter  students  to  realize  that  an  alternative  information  source  (e.g., 
a well-written  book  on  the  subject)  may  be  a more  efficient  use  of  their  time 
(although  I expect  them  to  be  able  to  tolerate  CAI/CM1I).  Thus  the  average  or 
below  average  group  may  accapt  more  readily  the  quality  of  instruction  they 
receive.  (Wexler) 


Great  consideration  should  be  given  to  designing  interface  devices  which  are 
adapted  to  humans  rather  than  forcing  humans  to  adapt  to  the  devices. 
Devices  which  are  designed  for  individuals  with  below  average  intelligence 
can  readily  be  used  by  individuals  of  normsi  intelligence  but  the  reverse  is 
not  necessarily  true.  By  designing  for  lower  intelligence  individuals  the 
terminals  will  be  available  to  a larger  audience.  This  is  especially  important 
in  the  military  where  a broad  spectrum  of  individual  differences  must  be 
accommodated.  (Halt) 
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No.  Terminal  should  be  flexible  enough  to  be  generally  useful.  The  cliche 
(with  a little  data  to  support  it)  is  that  bright  students  learn  no  matter  what 
the  instructional  treatment,  but  that  average  and  below  average  students 
need  special  attention.  However,  this  more  a problem  of  courseware 
development  and  implementation  priorities  than  of  terminal  design.  (Ford.) 


8 While  it  is  true  that  the  total  range  of  user  motivational  levels  must  ^ 

when  designing  CAI/CMI  user  interfaces,  do  you  agree  that  priority  should  be  given 

interfaces  intended  for  users  of  average  or  below  average  motivation.  Why. 


COMMENT:  Consensus  was  not  achieved  (6  Yes,  A No,  3 Other).  Representative 
responses  follov/: 


Yes,  the  high  motivation  trainees  will  acquire  information  and  ^1^'"® 
means  than  through  CAI  ar.i  CMI.  That  is,  computer  assistance  of  this  Kind  is 
less  important  for  highly  motivated  trainees.  Some  sKills  practice  which  is 
particularly  aided  by  computing  (e.g.,  highly  realistic  simulations)  may  be 
important  to  all  trainees,  and  particularly  the  motivated  ones  who  may  be 
expected  to  do  especially  well  on  the  job.  (Zinn) 


Motivation  is  a bag  of  worms.  Stay  out  of  it  in  considering  design  of  CAI/CMI 
user  interfaces.  Nobody  Knows  what  will  worK  with  a particular  student 
under  specified  conditions  at  any  particular  time.  (Ward) 


No  See  question  7.  Motivation  is  not  a simple  trait  --  different  students  are 
motivated  by  different  treatments.  Those  students  who  are  motivated  by 
CAI/CMI  should  probably  be  given  priority  for  usir’g  it.  (Ford) 
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9.  What  bottlenecks  make  it  difficult  for  users  of  average  Or  below  average  intelligence 
to  use  CAI/CMI  today? 


COMMENT:  Reliance  on  alphanumeric  rather  than  graphics  terminals  and  consequent 
emphasis  on  reading  comprehension  (Dean,  Giunti,  Kimberlin,  Sherwood,  Nawrocki). 
Poor  courseware  (Ditzik,  Merrill,  Sherwood).  Poor  learning  environments  (Bork,  Hall, 
Rockway,  Wexler,  Zinn),  e.g.. 


The  bottlenecks  reside  in  our  ignorance  about  what  things  are  difficult  and 
what  things  are  easy  for  users  of  average  or  below  average  intelligence. 
(Ford) 


Hardware  and  software  unreliability,  difficulty  of  use,  and  poor  human 
engineering.  (Rockway) 


Keyboard  arrangements  not  obvious  to  novice  user  of  keyboards. 
Identification  of  function  keys  confusing  to  novice  user.  Terminals  which  lack 
effective  pointing  capability  (light  pen,  cursor,  etc.)  Output  devices  (and 
displays)  with  limited  character  sets.  Inconvenient  editing  facility  for  altering 
text  before  input  (as  answer  or  request)  Frustration  of  slow  displays  and 
limited  line  length  (and  number  of  lines  on  screen)  (Zinn) 


10.  What  bottlenecks  make  it  difficult  for  users  of  average  or  below  average  motivation 
to  use  CAI/CMI  today? 


COMMENT:  Most  respondents  referred  to  response  for  question  9,  or  made  similar 
comments.  However,  the  concept  of  rewards  for  CAI/CMI  experiences  surfaced 
here: 


Bottlenecks  for  average  and  below  average  motivated  students  are:  A.  Their 
lack  of  experience  and  training  in  a self-paced  envirotiment  where  the  burden 
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is  placed  on  them  to  teach  themselves  rather  than  to  rely  on  a teacher  to  tell 
them  what  they  should  know.  B.  Their  inability  to  relate  current  success  to 
future  tasks.  C.  The  aloneness  a student  can  feel  m a ^ 

situation  D.  The  demotivating  environment  a military  student  might 
outside  the  classroom.  E.  The  possibilit/  of  an  unwelcome  assignment  after 
thp  r.ourse  is  comuleted.  (Kimberlin) 


11.  In  operational  terms,  what  does  an  "easy  to  use  terminal 
choice  in  options,  lack  of  an  alphanumeric  keyboard,  hardware  to 
aren’t  applicable). 


look  like?  (e.g.,  limited 
lock  out  features  that 


No  long  arrays  of  mysterious  buttons  with  cryptic  labels.  No  visible  controls 
which  the  user  should  not  use.  Off-on  switch  clearly  visible.  (Bork) 


Not  sure  that  this  is  a terminal  question  except  that  clutter  should  be  avoided 
and  operational  features  should  be  obvious  to  the  user  - i.e.,  see  a good 
office  copier  --  it  is  obvious  how  to  use  it.  (Dean) 


12  How  much,  when,  and  primarily  from  whom  should  the  beginning  user  learn  about  the 
■mterfacr?Tet  live  instructors,  other  osers,  remote  consultants,  software,  written  guides.) 


COMMENT:  Primarily  on-line,  from  the  system  itself  (Ford,  Hall,  Kimberlin,  Merrill, 
Nawrocki). 


Best  would  be  right  from  system  itself  (self-instruction)  via  terminal.  All 
Other  options  O.K.,  but  only  for  special  problems.fNawrocki) 
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COMMENT:  Primarily  instructors,  m rt,  experienced  students,  or  off-line  media  (Bork. 
Giunti,  Ward,  Zinr ) 


I don’t  favor  widespread  use  of  instructors  for  this  particular  task.  (Bork) 


At  the  initial  exposure,  live  instructors  should  provide  as  much  instruction  on 
interfaces  as  is  required  to  make  each  student  comfortabib.  (Giunti) 


Live  instructors  or  other  users,  primarily  because  this  is  the  method  they  are 
familiar  with.  (Ward) 


COMMENT:  Both  on-line  and  off-line  instruction  (Dean,  Ditzik,  Rockway,  Sherwood, 
Wexler). 


Absolute  basics  should  be  presented  by  a human,  and  aaditional  instruction 
should  be  given  by  the  device.  (Sherwood) 


I rinted  guides  plus  CAI  at  the  terminal  should  satisfy  most  learning 
requirements.  A human  proctor  or  instructor  should  be  available  for 
consultation,  either  in  person  or  via  communication  link.  (Rockway) 


13.  How  much,  when  and  primarily  from  whom  should  the  experienced  user  learn  about 
additional  aspects  of  the  interface? 


COMMENT:  Most  respondents  indicated  on-line  helps,  consultants,  and/or  off-line 
documentation. 
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From  on-line  and  off-line  documentation,  and  by  on-line  and  off-line 
communication  with  experienced  users.  (Sherwood) 


Manuals,  perhaps  video  or  slide-tape  presentation,  primarily.  Often  help  (or  a 
suggestion)  may  come  from  another  user.  For  complex  tasks,  human  aid 
(experienced  colleague  or  a live  instructor)  is  important.  (Zinn) 


14.  What  sorts  c.f  things  should  the  system  be  able  to  take  care  of  so  that  the  user  does 
not  have  to  learn  about  them  (e.g.,  bad  telephone  connection,  spelling  errors,  error 
recovery.) 


COMMENT:  Respondents  seemed  to  take  the  word  "system"  in  this  question  to  heart. 
For  example. 


The  system  should  include  all  the  resources  (even  human)  to  take  care  of  the 
problems  identified.  (Rockway) 


COMMENT:  While  r.jst  respondents  seemed  in  basic  agreement  with  Dr.  Rockway, 
Dr.  Wexler  added  some  details: 


The  system  should  handle  hardware  errors  and  shou'd  indicate  (e.g.,  by 
turning  on  a red  light)  that  it  is  trying  to  do  something.  The  light  is  turned 
off/to  green  when  the  difficulty  is  resolved.  A prolonged  red  might  lead  a 
user  to  try  the  CONNECT  - SIGN-ON  - RESTART  sequence.  Softer  errors 
remain  in  the  user’s  baliwick  (e.g.,  spelling,  referencing  an  unknown  file,  etc), 
although  the  system  should  try  to  make  plausible  guesses  about  the  user’s 
intent  and  indicate  its  hypothesis  prior  to  carrying  out  the  action.  (Wexler) 
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15.  When  things  are  not  functioning  properly  and  the  system  cannot  assist  the  user,  who 
and  what  are  the  primary  sources  of  support  that  the  user  can  fall  back  on. 


COMMENT:  Most  respondents  said  support  staff,  as  in  the  following  comments: 


A fall  back  program  should  consist  of  a prepared  package  of  study  guides  and 
references  that  relate  to  each  lesson  or  portion  of  the  lesson  that  is  on  the 
computer.  This  may  be  considered  redundant,  but  the  same  package  along 
with  the  off-line  material  will  be  a self-paced  course  program  that  may  serve 
after  a student  has  left  the  school,  or  serve  in  those  areas  where  a terminal 
is  not  available.  The  "who"  may  be  several  different  people,  depending  upon 
the  situation.  In  a unit,  it  may  be  a supervisor,  peer,  or  in  the  worse  case, 
the  man  may  have  to  "dig"  himself  out.  In  a formal  training  environment,  it 
will  be  the  class  instructor.  (Kimberlin) 


The  system  should  have  a manual  backup  in  the  form  of  instructor  support 
and  some  conventional  media  to  handle  automated  system  failures.  (Rockway) 


When -the  terminal  environment  becomes  a malfunctioning  environment  (Your 
terminal  is  working,  why  isn’t  mine?")  the  normal  sources  of  support  would  be: 

1.  A telephone  call  to  the  center  (unanswered?) 

2.  Nearby  users  (unavailable?) 

3.  A written  terminal  guide.  (WexUr) 


This  se  jms  a training  decision,  not  a hardware  design  consideration.  But..... 
When  'ihe  system  fails  it  should  recover  t.  much  as  possible 
perh'ips  prompting  the  student  for  information  needed  to  restart.  When  the 
system  can’t  handle  this  but  is  still  live,  it  should  offer  the  trainee  information 
which  may  be  helpful  for  him  to  initiate  recovery.  When  all  else  fails,  a 

written  guide  of  "What  to  do  if " should  be  handy  to  the  terminal;  in  many 

cases  it  can  get  a user  going  again  with  minimum  delay  and  embarrassment. 
Then  human  aid  should  be  at  hand  when  all  else  fails'.  (Zinn) 
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16.  There  was  consensus  that  the  greatest  payoff  will  come  from  an  emphasis  upon 
student  learning  (Rather  that  the  total  range  of  research,  course  writing,  and  learning). 
Are  we  at  a stage  where  enough  is  known  about  course  writing  and  learning  so  that 
reliable  and  effective  courseware  can  be  developed? 


COMMENT:  No  consensus  was  reached  (8  Yes,  6 No).  Representative  responses 
follow: 


No,  much  more  study  must  be  done  in  the  areas  of  what  is  good  material  for 
CAI  lesson,  what  are  best  strategies,  what  is  best  for  different  types  of 
students,  what  authoring  techniques  in  teams  are  most  efficient  and 
productive.  This  area  has  no  firm  data  - everyone  seems  to  do  their  own  pet 
thing  and  ignore  all  other’s  efforts.  (Giunti) 


There  is  still  a great  deal  that  we  need  to  know  about  course  writing  and 
learning  so  that  reliable  and  effective  course  work  can  be  developed. 
Perhaps  one  strategy  of  accomplishing  this  is  to  develop  course  material  and 
then  examine  it  carefully  to  see  what  features  of  it  have  contributed  to  the 
learning.  Repertories  of  alternative  procedures  and  techniques  need  to  be 
developed  and  examined  in  view  of  learning  difficulties  encountered  by 
students  so  that  course  material  can  be  improved.  (Hall) 


If  the  other  consultants  think  we  are  they  are  very  naive  as  to  what  is  known 
about  instruction.  I feel  we  have  a tremendous  amount  left  to  learn.  We 
have  just  started  to  learn  how  to  design  courses,  most  are  very  weak.  We 
have  many  more  questions  than  answers.  If  someone  thinks  we  know  it  all,  I 
invite  them  to  write  for  a list  of  questions  I’d  like  answered.  (Merrill) 


Ha!  Depends  who  you  talk  to!  If  we  knew  what  was  "effective"  and  for  whom 
and  under  what  conditions,  could  probably  develop  a major  in  "Educational 
Engineering"!  Despite  all  the  handbooks  and  texts,  courseware  development 
remains  heavily  intuitive.  Dialogue  systems  where  student  selects 
instructional  style  and  material  (Socratic)  will  impact  heavily  on  solving  this 
problem.  (Nawrocki) 


I think  so,  although  it  is  unlikely  that  good  courseware  can  be  proven  to  be 
good  by  actual  measurement  --  there  are  too  many  variables.  (Sherwood) 
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Yes  - especially  in  the  sophisticated  instructional  u'evelopment  models  used  by 
the  military.  See  Army  Regulation  350-100-1  (1968  version)  - Systems 
Engineering  and  Training  for  example.  (Ward) 


No,  not  in  general,  although  some  on-going  projects  may  take  exception.  It  is 
very  well  to  specify  a set  of  terminal  behaviors  in  a course  and  then  try  to 
carefully  structure  the  course  to  achieve  them.  However  I seem  to  remain 
unimpressed  with  the  quality  of  objectives  or  rather  the  "interpretations"  that 
have  been  made  of  them.  On  the  other  hand,  by  a suitable  redefinition  of 
competency  level,  much  existing  courseware  could  certainly  be  deemed 
"reliable  and  effective."  (Wexler) 


17.  During  the  first  round,  a lot  of  questions  about  this  versus  that  priority  were  asked. 
A number  of  you  felt  this  was  unfair,  unnecessary,  or  could  not  be  done.  In  this  round  we 
have  continued  extracting  your  priorities,  preferences,  and  opinions  about  areas  where 
payoffs  are  most  likely.  By  now  you  probably  have  a personal  set  of  the  most  central 
priorities  when  thinking  about  the  1980-85  military  CAI/CMI  user  interface.  What  are 
they? 


COMMENT:  Although  most  responses  to  this  question  repeated  responses  to 
previous  questions,  Giunti  and  Kimberlin  emphasized  transportability  as  a central 
priority: 


I would  also  like  to  see  a central  point  for  preparation  of  common  lesson 
material  so  that  all  Army  CAI/CMI  schools  would  teach  or  use  identical 
programs  for  instruction.  This  would  of  course  require  a certain  amount  of 
commonality  or  transportability  between  systems.  (Giunti) 


Central  priorities  revolve  around  transportability  of  lesson  material.  I don’t 
believe  that  CAI/CMI  will  really  be  able  to  grow  into  an  accepted  media  unless 
we  are  able  to  reduce  the  cost.  One  way  of  course  is  to  spread  the  cost 
over  many  students  and  this  means  distribution.  Therefore,  such  items  as 
language  and  terminal  standardization  will  provide  an  early  step  down  the 
road  to  transportability.  (Kimberlin) 
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Appendix  lll.E 

ROUND  THREE  QUESTIONNAIRE 
Your 

Name 

Date 

Please  return  to: 

T.  Martin  or  M.  Stanford 
Annenberg  School  of  Communications 
University  of  Southern  California 
Los  Angeles,  Ca.  90007 

GENERAL  QUESTIONS 


A number  of  you  remarked  in  your  round  two  responses  that  you  felt 
terminal-related  problems  were  not  the  area  where  a concerted  push  by  the 
military  would  result  in  the  greatest  payoff. 


1,  Considering  the  1980-85  timeframe,  how  would  you  rank  the  following 
investment  strategies  in  terms  of  their  potential  (1  = greatest  potential,  4 = 
least  potential)  for  advancing  CAI/CMI  state  of  the  art? 


invest  in: 

innovative  CAI/CMI  terminal  technology 

innovative  CAI/CMI  pedagogical  software 

technology  (see  the  pedagogical  software 
needs  table  below) 

^innovative  CAI/CMI  coursewriting 

large  scale  use  of  existing  hardware, 

software,  and  courseware. 
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2.  Do  you  think  that  it  is  in  the  best  interests  of  advancing  the  CAI/CMi  terminal 
state-of-the-art  to  combine  civilian  and  military  needs  as  opposed  to  focusing  on  just 
military  or  just  civilian  needs. 


COMBINE KEEP  DISTINCT OTHER 


3.  In  what  respects  do  you  feel  that  1980-85  military  CAI/CMI  terminals  should  differ 
from  civilian  ones?  (You  might  want  to  refer  to  the  table  of  terminal-related  functional 
needs  below.) 
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TERMINAL-RELATED  QUESTIONS 


4.  In  the  responses  to  the  second  round,  many  features  were  mentioned  as  relevant  to 
CAI/CML  While  it  is  natural  to  talk  in  terms  of  features,  our  terminal  experts  would  nfiuch 
rather  we  pin  you  down  on  the  functional  needs  underlying  various  types  of  features, 
leaving  final  feature  specifications  to  them.  Consequently,  we  have  developed  a list  cf 
functionai  needs.  They  are  listed  down  the  left  side  of  the  following  table.  For  each 
functional  need,  we  would  like  three  pieces  of  information  from  you: 


(1)  do  you  think  all  CAI/CMI  terminals  should  provide  for  the  need? 


++  definitely  yes 
+ would  be  nice 
0 neutral 

- probably  not 

— definitely  not 

(put  one  of  these  five  codes  in  each  row  of  the  ALL  TERMINALS  column.) 


(2)  assuming  that  not  all  terminals  provide  for  the  need,  what  special  situations 
justify  the  putting  together  of  special  terminals?  (put  situation-justification 
explanations  in  relevant  rows  of  the  EXCEPTION  column.) 


(3)  how  would  you  restate  the  functional  need  in  terms  that  get  closer  to  what  you 
perceive  the  real  need  to  be?  (,'<ut  rewordings  in  each  row  of  the  REWORDED  NEED 
column.) 


Keep  track  of  additional  need  categories  that  occur  to  you  since  questions  5 and  6 will  ask 
for  them. 
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TRANSIENT  VISUAL  OUTPUT 


a.  A variety  of  predefined  symbol 
sets  (3  or  more)  can  be  intermixed 
during  display. 


b.  Special  symbol  sets 
programmable  when  needed. 


are 


c.  Simple  straight  line  figures 

(involving  few  lines)  can  be 
generated. 

d.  Complex  line  figures  (involving 

many  lines)  can  be  generated. 

e.  Complex  figures  (involving 
shading  and  texture)  can  be 
generated. 

f.  Displays  may  be  generated  in 

color. 


g.  Displays  may  be  generated  that 
contain  as  many  as  four  thousand 
readable  symbols. 

h.  Stored  visuals  (with  resolution 
equal  to  that  of  a television  screen) 
can  be  displayed. 

i.  Stored  visuals  (with  resolution 
high  enough  for  reading  a picture  of 
a college  textbook  page)  can  be 
displayed. 

j.  Moving  visuals  can  be  displayed  at 
flicker-free  speeds. 
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HARDCOPY  VISUAL  OUTPUT 


K.  Exact  reproductions  of  screen 
images  can  be  produced. 


I.  Alphanumeric  text,  using  a single 
type  font,  can  be  printed. 


TRANSIENT  AUDIO  OUTPUT 


m.  Computer-composed  speech  can 
be  generated  and  transmitted  to  the 
user. 


n.  Pre-recorded  audio  output  (equal 
in  quality  to  AM  music)  can  be 
transmitted  to  the  user. 


OTHER  OUTPUT 


0.  Signals  are  available  for 
controlling  external  equipment. 


p.  Lights  under  some  keys  on  the 
keyboard  can  be  turned  on  or  off. 
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INPUT 

q.  Strings  of  alphanumeric 
characters  can  be  typed  in  by  the 
user. 

r.  Frequently  invoked  functions  can 
be  specified  unambiguously  by 
carrying  out  a single  action. 

s.  Locations  on  the  screen  can  be 
specified  by  touching  or  pointing. 

t.  Lines  can  be  drawn  on  a 
two-dimensional  surface. 

u.  A clearly  spoken  word  chosen 
f.  om  a limited  vocabulary  (about  20 
words  at  any  one  time)  can  be 
recognized  about  907.  of  the  time. 

V.  Signals  that  are  equivalent  to 
terminal-entered  input  can  be 
received  from  devices  plugged  into 
the  terminal. 

OTHER  TERMINAL  FEATURES 

w.  The  stand-alone  terminal  has 
sufficient  processing  capability  to 
manage  instruction  707.  of  the  time. 

X.  Digital  and  analog  memory  are 
available  within  the  terminal. 
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5 Now  that  you  have  seen  the  twenty-four  functions,  what  additional  needs  come  to  mind 
for  which  you  strongly  feel  features  should  be  standard  across  all  terminals. 


6.  What  additional  features  do  you  feel  should  be  developed  for  special  situations?  Again 
we  would  appreciate  situation-justification  explanations. 
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PEDAGOGICAL  SOFTWARE  QUESTIONS 

7.  In  the  responses  to  the  second  round,  many  of  you  mentioned  that  there  was  a need 
for  a more  adaptive,  intelligent  interplay  between  student  and  courseware  or  teacher  and 
coursewriter.  Since  there  are  so  many  things  you  might  have  had  in  mind,  and  since  it 
appeared  so  frequently,  we  felt  it  was  necessary  to  probe  the  area  in  greater  depth.  A 
list  of  software  functional  needs  has  been  developed  similar  to  the  terminal  functional 
needs  list.  This  time  we  are  asking  for  four  pieces  of  information  about  each  need: 

(1)  assuming  that  by  1980-85  it  is  possible,  do  you  think  all  CAI/CMI  systems  should 
provide  for  the  need? 


++  definitely  yes 
+ would  be  nice 
0 neutral 
- probably  not 
" definitely  not 

(put  one  of  these  f>ve  codes  in  each  row  o^  the  ALL  SYSTEMS  column.) 


(2)  assuming  that  by  1980-85  it  is  possible,  but  that  not  all  systems  provide  for  the 
need,  what  special  situations  justify  the  putting  together  of  special  software?  (put 
situation  - justification  explanations  in  relevant  rows  of  the  EXCEPTION  column.) 


(3)  how  would  you  restate  the  functional  need  in  terms  that  get  closer  to  what  you 
perceive  the  real  reed  (or  needs)  to  be?  (put  rewordings  in  each  row  of  (he 
REWORDED  NEED  column.) 


(4)  how  great  a payoff  do  you  anticipate  from  an  investment  now  in  software  that 
will  respond  to  the  need  by  1980-85? 


++  very  high  payoff  anticipated 
+ high  payoff 
0 moderate  payoff 

- low  payoff 

— no  payoff  anticipated 


f-1" 


r I 


STUDENT-RFIATED  SOFTWARE 
The  system  can  synthesize: 


a.  Instructional  sequences  that  are 
tailored  to  the  abilities  and/or 
weaknesses  of  individual  students. 


b.  Problems  and  examples  that 
respond  to  the  interests  of  particular 
students. 


c.  Hints  that  reduce  the  difficulty  of 
problems. 


d.  Summaries  of  a student’s 
progress  throughout  the  course. 


The  :y.  tern  can  '^ollow  and  respond 
meaningfully  to: 


e.  Course-related  problem-solving 
strategies  (even  though  novel) 
employed  by  students. 


f.  Course-independent  (but 

pedagogically  relevant) 

problem-solving  strategies  employed 
by  students. 


g.  Pauses  due  to  a student’s  inability 
to  decide  what  to  do  next. 


h.  Course-related  questions  or 
statements  typed  in  by  students. 
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i.  Course-independent  (but 
pedagogically  relevant)  questions 
typed  in  by  students. 

j.  Dialog  cues  (i.e.,  topic  shifts, 
impatienc'i,  bewilderment,  wandering 
attentiun). 

TEACHER-RELATED  SOFTWARE 
The  system  can: 

k.  Derive  course-related  strategies 
from  examples  provided  by  the 
instructor. 

l.  Accumulate  course-reiated 
concepts  (with  associated  vocabulary) 
from  examples  provided  by  the 
instructor. 

m.  Contrast  concept/strategy 

information  with  information  about 
potential  students,  detect  difficulties, 
and  advise  the  instructor  of  the 
difficulties. 

n.  Discover  patterns  of 
course-related  behavior  and  advise 
the  instructor  during  the  course. 
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8.  Now  that  you  h'jve  seen  the  fourteen  software  functions,  what  additional  needs  come> 
to  mind  for  which  you  strongly  feel  features  should  be  standard  across  all  systems? 


9.  What  additional  software  features  c you  feel  should  be  developed  for  special 
situations?  Again  we  would  appreciate  situation  - justification  explanations. 


- --- 
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Appendix  III.F 
ROUND  THREE  SUMMARY 


1.  Considering  the  1980-85  time  frame,  how  would  you  rank  the  following  investment 
strategies  in  terms  of  their  potential  (1  » greatest  potential,  4 “ least  potential)  for 
advancing  CAi/CMI  state  of  the  art? 

Invest  in: 

1 1 1 1 1 1222xx34  Innovative  CAI/CMI  pedagogical 
software  technology 

1 1 1 1122223334  Innovative  CAI/CMI  coursewriting 

1122x33444444  Innovative  CAI/CMI  terminal  technology 

22x3333344444  Large  scale  use  of  existing  hardware 
software,  and  courseware 
(x  stands  for  an  averaged  rating  of  2.5) 


2.  Do  you  think  that  it  is  in  the  best  interests  of  advancing  the  CAI/CMI  terminal 
state-of-the-art  to  combine  civilian  and  military  needs  as  opposed  to  focusing  on  just 
military  or  just  civilian  needs. 


Combine  —lO—  Keep  Distinct  —0—  Other  —2—  No  Response  --2— 


(The  two  "other"  responses  indicated  that  needs  could  be  combined  except  that  in  some 
situations  special  terminals  would  be  needed.  The  comments  have  therefore  been 
incorporated  into  the  responses  to  question  inree.) 


3.  In  what  («.'*pects  do  you  feel  that  1980-85  military  CAI/CMI  terminals  should  differ 
from  civilian  ones? 

No  differences  4 

No  response  3 

Rugged  construction  to  enable  use 
in  hostile  environments  5 

Greater  emphasis  on  nonverbal 
communication  (ie.  audio, 
video,  graphics,  non-keyboard 
input)  3 

Greater  need  for  stand-alone  systems  1 
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4(1).  Do  you  think  all  CAI/CMI  terminals  should  provide  for  the 
following  functional  needs? 


++  definitely  yes  (shown  below  as  it) 
+ would  be  nice 
0 neutral 

probably  not 

--  definitely  not  (shown  below  as  ■) 
no  response  (shown  below  as  o) 


[The  sections  below  were  reordered  in  an  informal  order  of 
positiveness  of  response,  for  the  participants’  ease  of 
reading,  as  shown.) 


iticicicitititititititit++ 

ici(i(i(itititi(ititit++Q 


Strings  of  alphanumeric  characters 
can  be  typed  by  the  user. 

Frequently  invoked  functions  can  be 
specified  unambiguously  by  carrying 
out  a single  action. 


i<i<i<i<i<i<i(i<iti(++ + ■ 

b.  Special  symbol  sets  are 
programmable  when  needed. 

VW’n'o'o'o'o'oV +++00* 

a.  A variety  of  predefined  symbol 

sets  (3  or  more)  can  be  intermixed 
during  display. 

i<i<i<i<ifififif++ +00" 

c.  Simple  straight  line  figures 

(involving  few  lines)  can  be  generated. 

>VVf  + ++ + 0 0 - 

0.  Signals  are  available  for 

controlling  external  equipment 

VfVn'f**Vif>V++++00- 

s.  Locations  on  the  screen  can  be 
specified  by  touching  or  pointing. 

*>v**Vf++++++0 — V.  Signals  that  are  equivalent  to  terminal- 

entered  input  can  be  received  from 
device^*  plugged  into  the  terminal. 
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*****+++++ 00 • ■ 

h.  Stored  visuals  (with  resolution  equal  to 
that  of  a television  screen)  can  be 
displayed. 

****>’(*++++0--” 

1.  Alphanumeric  text,  using  a single  type 
^ont,  can  be  printed. 

***++++++++00= 

t.  Lit'.es  can  be  drawn  on  a two-dimensional 
surface. 

>V  ***Vf+++++0  — ■ 

d.  Complex  line  figures  (involving  many 
lines)  can  be  generated. 

****++++++00-= 

i.  Stored  visuals  (with  resolution  high 
enough  for  reading  a picture  of  a 
college  textbook)  can  be  displayed. 

***++++++0000- 

w.  The  stand-alone  terminal  has 

sufficient  processing  capability 
to  manage  instruction  70%  of  the  time. 

****++++++0— 

n.  Pre-recorded  audio  output  (equal 
in  quality  to  Afl  music)  can  be 
transmitted  to  the  user. 

**+++++++++0— 

j.  Moving  visuals  can  be  displayed 
at  flicker-free  speeds. 

V<Vf++++'r++00G*“ 
*+++++++0000 — 


***++++000 

*+•*++++00000 — 
iV**+++++O0--«“ 


f.  Displays  can  be  generated  in  color, 
m.  Computer-composed  speech  can 
be  generated  and  transmitted 
to  the  user. 

p.  Lights  under  some  keys  on  the 
keyboard  can  be  turned  on  or  off. 

X.  Digital  and  analog  memory  are 
available  within  the  terminal, 
k.  Exact  reproductions  of  screen 
images  can  be  produced. 


*++++00000-= 


+++++++00 — • 


+++++00G0 — • 


e.  Complex  figures  (involving  shading  and 
texture)  can  be  generated, 
u.  A clearly  spoken  word  chosen  from 
a limited  vocabulary  (about  20  words 
at  any  one  time)  can  be  recognized 
about  90%  of  the  time, 
g.  Displays  can  be  generated  that  contain 
as  many  ns  four  thousand  readable 
symbols. 


4(2,3)  Questions  regarding  special  situations  justifying  special 
terminals  and  restated  functional  needs. 
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Seven  of  the  lA  respon-jents  made  no  comments. 

Of  the  remaining  seven,  two  or  more  people  made  the  same  comment 
only  three  times. 


Whether  or  not  terminals  or  software  systems  should  provide  the 
features  is  an  empirical  question  based  on  the  feature’s 
demonstrated  instructional  effectiveness  (or  lack  thereof). 

WARD,  NAWROCKI 


Primary  need  is  for  digital,  not  analog  memory. 

DEAN,  DITZIK 


Exact  reproductions  of  screen  images  can  be  produced  on  demand 
at  a single  station  within  the  terminal  area  to  serve  all  users 
- but  not  on  every  terminal. 

HALL,  DITZIK,  KIMBERLIN 


5 and  6 — Questions  regarding  additional  needs  and  features. 


Seven  of  the  14  respondents  made  no  comments. 

Of  the  remaining  seven,  in  no  case  did  two  or  more  people  suggest 
the  same  feature. 


7(1).  Do  you  think  all  CAI/CMI  systems  should  provide  for  the 
following  needs? 


++ 

+ 

0 


5 


definitely  yes  (shown  below  as  *) 
would  be  nice 
neutral 
probably  not 

definitely  not  (shown  below  as  -) 
no  response  (shown  below  as  o) 


[The  sections  below  were  reordered  in  an  informal  order  of 
Dositiveness  of  response,  for  the  participants’  ease  of  reading,  as 

shown. ] 

********v«v++oo  a.  Instructional  sequences  that  are 

tailored  to  the  abilities  and/or 
weaknesses  of  individual  students. 
******++++++00  b.  Problems  and  examples  that  respond 


I 
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***Vf+++++++++0 


t't  %'t  * ++++++0« 


****+++++++00- 


****+++++++00- 


***Vf++++++O00- 


* * * 'V + + + + + + 0 0 0 - 


*Vf**++++++00 — 


****+++++0000- 


**Vo'oV+OO000000 


VoV++++++O0000- 


*++++++++00000* 


*+++++000000-* 


to  the  interests  of  particular 
students. 

c.  Hints  that  reduce  the  difficulties 
of  problems. 

d.  Summaries  of  a student’s  progress 
throughout  the  course. 


m.  Contrast  concept/strategy 
information  with  information  about 
potential  students,  detect 

di  f f icul t i es, and  advise  the  instructor 
of  the  difficulties. 

n.  Discover  patterns  of  course- 
related  behavior  and  advise  the 
instructor  during  the  course. 

e.  Responsive  to  course-related 
problem-solving  strategies  (even 
though  novel)  employed  by  students. 

k.  Derive  course-related  strategies  from 
examples  provided  by  the  instructor. 

h.  Respond  to  course-related  questions 
or  statements  typed  in  by  students. 

l.  Accumulate  course-related  concepts 
(with  associated  vocabulary)  from 
examples  provided  by  the  instructor. 


g.  Respond  to  pauses  due  to  a student’s 
inability  to  decide  what  to  do  next, 
f.  Respond  to  course-independent  (but 

pedagogi ca I I y relevant)  problem-solving 
strategies  employed  by  students, 
j.  Respond  to  dialog  cues  (eg.  topic 
shifts,  Impatience,  bewilderment 
wandering  attention), 
i.  Respond  to  course- i ndependent  (but 
pedagogical ly  relevant)  questions 
typed  in  by  students. 
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7(2  and  3)  Questions  regarding  special  systems  justifying  special 
software  and  restated  functional  need;;. 

Nine  of  the  lA  made  no  comments. 

None  of  the  remaining  5 made  equivalent  comments. 


I 


7(A)  Question  regarding  payoff  ratings  for  software  development. 

Ratings  in  this  column  were  for  the  most  part  identical  to  7(1) 
ratings  and  zc  are  not  repeated  here. 

8 and  9 Questions  regarding  additional  needs  and  additional  software 
features. 

Nine  of  the  lA  made  no  comments. 

Two  of  the  remaining  5 mentioned  that  portability  across  systems 
is  also  highly  desirable. 

HALL,  RXKWAY 
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Navy  Parsonnal  Program  Support 
Activity 

Building  1304,  Bolling  RFB 
Uashington,  O.C.  30336 

Off  lea  of  Civilian  Nanpowar  Nanagamant 
Coda  64 

Uashington,  O.C.  20390 
RTTNi  Dr.  Richard  NIohaus 

Off  lea  of  Civilian  Nanpouar  flanagamant 
Coda  263 

Uashington,  D.C.  20390 

Off  lea  of  Naval  Rasarva 
Coda  3055 

Nau  Or'aans,  LR  70146 


Dr.  hart  in  Rockway  (RFHRL-TT) 

Loury  RFB 
Colorado  80230 

Hajor  P.J.  DaLao 
Instruct  I Tachnology  Branch 
PF  Human  Rasourcas  Laboratory 
Loury  RFB,  CO  80230 

Dr.  HI  trad  R.  Frag  I y 
RF03R-NL 

1400  U I Ison  Boulavard 
Rrlington,  Vfl  22209 

Capt.  Jack  Thorps,  USRF 
Flying  Training  Division 
RFHRL-FT 

Ul 1 1 lams  RFB,  RZ  85224 

RFHRL-PEO 
Stop  #63 

I ickland  RFB,  TX  7823F 

Dirac  tor.  Off  lea  of  Hanpouar 
Utl I Izatlon 

Haadquartars,  flarina  Corps  (Coda  tIPU) 
nCB  (Building  2009) 

Quant Ico,  VR  22134 

Dr.  R.L.  Slalkosky 
Sciantific  Rdvisor  (Coda  RD-1) 
Haadquartars,  U.S.  tiarina  Corps 
Uashington,  D.C.  20380 

Chlaf,  Academic  Department 

Education  Cantar 

Harlna  Corps  Davalopmant  and 

Education  Command 

Narine  Corps  Base 

Quant Ico,  VR  22134 

1 

Hr.  E.  R.  Dover 

2711  South  Valtch  Street 

Rrlington,  Vfl  22206 

I Hr.  Joseph  J.  Couan,  Chlaf 

Psychological  Rasaarch  Branch 
(G-P-1-62) 

U.S.  Coast  Guard  Headquarters 
Uashington,  D.C.  20590 


Dr.  H.  D.  Havron 
Human  Sciencas  Research,  Inc 
7710  Old  Spring  House  Road 
U'est  Gate  Industrial  Park 
HcLaan,  Vfl  22101 

HumRRD  Central  Division 
400  Plaza  Bui Iding 
Pace  Boulavard  at  Fairfield  Dr 
Pensacola,  FL  32505 

HumRRD-Uastern  Division 
27857  Baruick  Drive 
Carmel,  CR  93921 
RTTNi  Library 

HumRRD  Cant-«i  DIvislon-Columb 
Office 

Suita  23,  '^601  Cross  Country  D 
Columbus,  Gfl  31906 

HumRRD-Uestarn  Division 
27857  Baruick  Drive 
Carmel,  CR  93921 
RTTNi  Dr.  Robert  Vinabarg 

HumRRD 

Joseph  fl.  Austin  Building 
1939  Goldsmith  Lana 
Louisville,  KY  40218 

Dr.  Laurence  B.  .'ohnson 
Laurence  Johnson  8 Associates, 
2001  S 'treat,  N.U.,  Suita  502 
Uashington,  D.C.  20009 

Or.  Arnold  F.  Kanarick 
Honayuall,  Inc. 

2600  Ridge  Parku 
Hlnneapol is,  HN  55413 

Or.  Roger  R.  Kaufman 

U.S.  International  University 

Graduate  School  of  Humai.  Bahav 

El  I lott  Campus 

8655  E.  Pomarada  Road 

San  DIago,  CR  92124 
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Military  flMlatant  for  Human  Ratourcaa  Dr.  Stavan  U.  Kaala 

Dflica  of  tha  Sacratary  of  Dafanta  Univaralty  of  Dragon 

Chlaf  of  Naval  Dparatlona  Room  3D129,  Pantagon  Dapartmant  of  Psychology 

DP-967P7  Uashlngton,  D.C.  28381  Eugana,  DR  97483 

Uashington,  D.C.  28358 

RTTNi  CRPT  H.J.M.  Connary  Rdvancad  Raaaarch  Projacta  Rgancy  Dr.  David  Klahr 


Supar  Intandant 

Naval  Poatgraduata  School 

Montaray,  CR  93948 

Mr.  Gaorga  N.  Gralna 
Naval  Saa  Syatams  Command 
SER  847C12 

Uashington,  D.C.  28362 

Chlaf  of  Naval  Tachnical  Training 
Naval  Rlr  Station  Mamphia  (7S) 

Ml  1 1 Ipgton,  TN  38854 
RTTNi  Di  . Noriaan  J.  Karr 

Commanding  Dfficar 
Sarvlca  School  Command 
U.S,  Na'<al  Training  Cantor 
San  Olago,  CR  92133 
RTTNi  Coda  83838 

Principal  Civilian  Rdvisor 
for  Education  and  Training 
Naval  Training  Command,  Coda  88R 
Ponsacola,  FL  32588 
RTTNi  Dr.  Ullllam  L.  Maloy 

Diractor 

Training  Rnalyala  8 Evaluation  Group 
Coda  N-88T 

OapartMnt  of  tha  Navy 
Orlando,  FL  >2813 
RTTNi  Or.  RIfrad  I'.  Smoda 

Chlaf  of  Naval  Training  Support 
Coda  N-21 
Bui Iding  45 
Naval  Rlr  Stalton 
Panaacola,  FL  32588 

LCDR  C.  F.  Logan,  USN 
r-14  Managamant  Systam 
COMKITREUUINGPRC 
NRS  Miramar,  CR  92145 

Navy  Parsonnal  Rasaarch 
and  Davalopmant  Cantor 
Coda  1 

San  Diago,  CR  92152 


Rdminlatrat i va  Sarvicaa 
1488  Ullson  Boulavard 
Rrlington,  VR  22289 
RTTN:  Rrdalla  Hollouay 

Dr.  Harold  F.  O'Nall,  Jr. 

Rdvancad  Rasaarch  Projacts  Rgancy 
Human  Resourcas  Rasaarch  DIfica 
1488  Ullson  Boulavard 
Rrlington,  VR  22289 

Or.  Robart  Yeung 

Rdvaitcad  Research  Projacts  Rgancy 
Human  Resourcas  Rasaarch  Dffica 
1488  Ullson  Boulavard 
Rrlington,  VR  22289 

Dr.  Lorralna  D.  Eyda 

Partonial  Rasaarch  and  Davalopmant 

Cantor 

U.S.  Civil  Sarvlca  Commisalon 
1988  E Straat,  N.U. 

Uashington,  D.C.  28415 

Or.  Ullllam  Gorham,  Diractor 
Parsonnal  Rasaarch  and  Davalopmant 
Cantor 

U.S.  Civil  Sarvlca  Commission 
1988  E Straat,  N.U. 

Uaahington,  D.C.  28415 

Dr.  Varr  Urry 

Paraonnal  Raaaarch  and  Davalopmant 
Cantor 

U.S.  Civil  Sarvlca  Commlaalon 
1988  E Straat,  N.U. 

Uashington,  D.C.  28415 

Dr.  Erik  McUllliams,  Diractor 
Tachnological  Innovations  in 
Education  Group 
National  Sclanca  Foundation 
1888  G Straat,  N.U.,  Room  U 658 
Uashington,  D.C.  28558 

Dr.  Richard  C.  Rtklnson 
Oaputy  Diractor 
National  Sclanca  Foundation 
1888  G Straat,  N.U. 

Uashington,  D.C.  28558 


Carnagia-Mal Ion  Univaralty 
Dapartmant  of  Psychology 
Pittsburgh,  PR  15213 

Dr.  Ezra  S.  Krandal 
Uni  varsity  of  Pannsylvania 
Uharton  School,  DH-CC 
Phi ladalphia,  PR  19174 

Dr.  RIma  E.  Lantz 
University  of  Danvar 
Dar.>r  Rasaarch  Inatituta 
Industrial  Economics  Divislo 
Denver,  CD  88218 

Mr.  Brian  McNa I ly 
Educational  Tasting  Service 
Princeton,  NJ  88548 

Dr.  Robert  R.  Mack  la  | 

Human  Factors  Research,  Inc 
6788  Cor  ton  Drive 
Santa  Barbara  Rasaarch  Park 
Golata,  CR  93817 

'i 

Dr.  Lao  Munday,  Vice  Prasidan 
Rmorican  Collage  Tasting  Prog 
P.D.  Box  168 
Iowa  City,  IR  52248 

Mr.  R.J.  Pasch,  Praaldant 
Eclactach  Rssoclataa,  Inc. 

P.D.  Box  178 

North  Stonington,  CT  86359 

Mr.  Luigi  Patrul  lo 
2431  North  Edgauood  Straat 
Rr  I Ington,  VR  22287 

Dr.  Stavan  M.  Pina 
University  of  Minnesota 
Dapartmant  of  Psychology 
Minneapolis,  MN  55455 

Dr.  Diane  M.  Ramsay-Klaa 
R-K  Rasaarch  8 Systam  Dasig 
3947  RIdgamont  Drive 
Malibu,  CR  98265 
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Navy  Ptrtonnti  Research  and 
Development  Center 
Code  2 

San  Diego,  CR  921S2 
RTTNi  fl.  fl.  Sjoholm 


Navy  Personnel  Research 
and  Development  Center 
Cods  306 

San  Diego,  CR  921S2 
RTTNi  Dr.  J.  H.  Stelnemann 


Dr.  Rndrett  R.  Oolnar 
Technological  Innovations  In 
Education  Croup 
National  Science  Foundation 
1000  C Street,  N.U. 
Uashington,  D.C.  20S50 


Navy  Personnel  Research 
and  Development  Center 
San  Diego,  CR  921S2 
RTTNi  Library 


Dr.  ttarshall  S.  Smith 
Rssistant  Rcting  Director 
Program  on  Essential  Skills 
National  Institute  of  Education 
Broun  ’’j tiding.  Room  815 
19th  and  11  Streets,  N.U. 
Uashington,  D.C.  29208 


Navy  Personnel  Research  and 
Oavelopmant  Center 
Code  9041 

San  DIago,  CR  92152 
RTTNi  Dr.  J.D.  Fletcher 


U.S.  Civil  Service  Commission 
Federal  Office  Building 
Chicago  Regional  Staff  Division 
Regional  Psychologist 
230  South  Dearborn  Street 
Chicago,  IL  69604 
RTTNi  C.S.  Ulnieulcz 


D.  n.  Gragg.  CRPT,  NC,  USN 

Head,  Educational  Programs  Development 

Dapartmant 

Naval  Health  Sciences  Education  and 
Training  Command 
Bathasda,  HD  20014 


Dr.  Carl  Frederiksen 
Learning  Division,  Basic  Skills  Croup 
National  institute  of  Education 
1200  19th  Street,  N.U. 

Uashington,  D.C.  29208 


Technical  Director 
U.S.  Rrmy  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1368  Ul Ison  Blvd. 

Rrlington,  VR  22209 


Dr 


Scarvla  B.  Rnderson 
Educational  Testing  Service 
17  Executive  Park  Drive,  N.E. 
Rtlanta,  CR  30329 


Armed  Forces  Staff  College 
Norfolk,  VR  23S11 
RTTNi  Library 


Or.  John  Rnnett 
Department  of  Psychology 
The  University  of  Uaruick 
Coventry  CV47RI 
ENGLAND 


CosNsandant 

U.S.  Rrmy  Infantry  School 
Fort  Banning,  CR  31905 
RTTNi  RTSH-Or-T 


hr.  Samuel  Ball 
Educational  Testing  Service 
Princeton,  N.J.  08540 


Dsputy  Commander 

U.S.  Rrmy  Institute  of  Rdministrat Ion 
Fort  Benjamin  Harrison,  IN  46216 
RTTNi  ER 


Dr.  Gerald  V.  Barrett 
University  of  Rkron 
Department  of  Psychology 
Rkron,  OH  44325 


Dr.  Joseph  U.  Rignay 
University  of  Southern  Cal  I for 
Behavioral  Technology  Laborato 
3717  South  Grand 
Los  Rngelas,  CR  90007 


Dr.  Leonard  L.  Rosenbaum,  Chal 
hontgomery  College 
Department  of  Psychology 
Rock V I lie,  HD  29859 


Dr.  George  E.  Rouland 
Rouland  and  Company,  Inc. 
P.O.  Box  61 
Haddonflald,  NJ  08033 


Dr.  Rrthur  I.  Siegel 
Rpp I lad  Psychological  Ssrvic 
404  East  Lancaster  Rvanua 
Uayna,  PR  19987 


Dr.  Richard  Snou 
Stanford  UnivarsI ty 
School  of  Education 
Stanford,  CR  9430S 


Or.  C.  Harold  Stone 
1428  Virginia  Rvenue 
Glendale,  CR  91202 


hr.  Dennis  J.  Sullivan 

c/o  HRISC,  Building  119,  U.S. 

P.O.  Box  90515 

Los  Rngeles,  CR  99999 


Dr.  Patrick  Suppes 
Stanford  University 
Institute  for  flathematicai  Stu 
in  the  Social  Sciences 
Stanford,  CR  94395 


Dr.  Benton  J.  Underwood 
Northuestern  University 
Department  of  Psychology 
Evanston,  IL  60201 


Or.  Carl  R.  Vest 
Batteiie  hemorlai  Institute 
Uash I ng  ton  Opera  1 1 ons 
2030  n Street,  N.U. 
Uashington,  D.C.  20936 
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Op.  Frank  J.  Harr  1 1 

U.S.  Army  Rtitarch  Inatituta  <or  tha 

Bthavioral  and  Social  Sciancti 

1361  Uliion  Bouiavard 

Arlington,  VA  22269 


Or.  Barnard  n.  Basa 
UnIvarsity  o<  Rochactar 
Graduata  School  t><  tlanagemant 
Rochastar,  NY  14627 


Or.  Stanlay  L.  Cohan 

U.S.  Army  Raiaarch  Initltuta  lor  lha 

Bahavloral  and  Social  Sclancas 

1366  Ml  Ison  Bouiavard 

Arlington,  VA  22269 


Or.  Ronald  P.  Carvar 

School  ol  Education 

Unlvarslty  ol  Ilistoiiri-Kantas  City 

5168  Rockhi 1 1 Road 

Kansas  City,  110  64116 


Dr.  David  J.  Uaiss 
Unlvarslty  ol  llinnasota 
Oapartmant  ol  Psychology 
N666  El  I iott  Hail 
llinnaapoils,  tIN  55455 


Dr.  Anl ta  Uast 
Oanver  Rasaarrh  Inst  I tuts 
Unlvarslty  ol  Oanvar 
Donvar,  CO  86218 


Dr.  Ralph  Ousak 

U.S.  Army  Rasaarch  Instituta  lor  tha 
Bahavloral  and  Social  Sclancas 
1366  UllBon  Bouiavard 
Arlington,  VA  22269 


Cantury  Rasaarch  Corporation 
4113  Laa  Highway 
Arlington,  VA  22267 


Dr.  A.  Charras 
BEB  512 

Unlvarslty  ol  Taxas 
Austin,  TX  78712 


Or.  Kannath  N.  Uaxiar 
Unlvarslty  ol  Calilornla 
School  ol  Social  Sclancas 
Irvina,  CA  92664 
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